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BLOCK  IV. 1:  IMPLEMENT  INSTRUCTIOIAL  MANAGEMENT  PLAN 
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OVERVIEW 


This  block  provides  guidelines  for  carrying  out  the  planning  and 
classroom  management  functions  essential  to  successful  Implementation 
of  an  instructional  program.  The  emphasis  here  is  on  assuring  that 
the  right  people  are  at  the  right  place  at  the  right  time,  that  they 
know  what  they  are  supposed  to  do,  and  that  they  have  the  materials, 
equipment,  and  facilities  to  do  it. 
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1W’L»CNT  INSTRUCTIONAL  HAtlAGEMtNT  PLAN 


1.0  INTIWOUCTION 


t«I 


Following  the  validation  and  revision  cycle  In  Block  III. 5.  the 
instructional  program  Includes: 

1.  What  you  wish  to  accomplish  (In  the  form  of  learning 
(^Jectlves) , 

2.  How  you  Intend  to  accortpllsh  It  (in  the  form  of  an  Instruc- 
tional management  p1an»  delivery  system,  and  instructional 
materials),  and 

3.  How  you  will  know  If  you  meet  your  objectives  (In  the  form 
of  tests  and  other  appraisal  instruments). 

Now  the  time  has  come  to: 

1.  Find  out  If  the  Instruction  and  management  plan  work  in  a 
field  setting. 

2.  If  not,  find  out  where  they  do  not  work,  and 

3.  Revise  until  personnel  and  materials  are  developed  to  the  point 
that  students  meet  the  learning  objectives. 

As  inputs  to  this  block,  everything  that  has  been  done  in  Phases  I,  II, 
and  III,  plus  the  internal  evaluation  plan  that  will  be  discussed  in  Block 
V. 1 are  available.  You  may  be  surprised  that  you  need  an  input  from  Block  V.l 
when  you  are  just  beginning  Block  IV. 1.  Actually,  the  internal  evaluation 
discussed  in  Block  V.l  consists  of  three  major  efforts.  These  are: 

1.  Prepare  evaluation  plan, 

2.  Collect  evaluation  data,  and 

3.  Analyze  evaluation  data  and  make  recommendations. 
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Item  1 must  take  place  prior  to  beginning  Phase  IV  of  the  ISD 
Hodel.  Item  2 Bust  take  place  during  the  conducting  of  Instruction 
in  Block  IV. 2.  And,  Item  3 cannot  be  completed  until  after  Phase  IV 
is  ccNnpleted. 

To  further  co<n)11cate  things,  the  inf)lementation  of  the  instructional 
management  plan  in  this  block  begins  before  the  conducting  of  instruction 
in  the  next  block,  but  is  not  complete  until  the  next  block  is  cotrplete. 

A rough  idea  of  the  time  relationships  between  these  blocks  is  shown  in 
Figure  IV. 1.  You  should  not  assume  from  this  that  these  steps  in  the  ISD 
model  take  place  only  one  tin«.  If  there  are  problems  with  the  program, 
all  three  blocks  will  have  to  be  repeated  until  the  problems  are  identi- 
fied and  corrected. 

The  Individuals  involved  In  Phases  I,  II,  and  III  may  or  may  not  be 
the  instructors,  supervisors,  or  classroom  managers  who  conduct  the 
instruction. 


Block  IV. 1: 

Inclement 

Instructional 

Management 

Plan 

Block  IV. 2 

Conduct 

Instruction 

Block  V.l: 
Conduct 
Internal 
Evaluation 


Begin POINT  IN  TIME End 


FIGURE  IV. 1:  Approximate  Points  in  Time  When  Several 
ISD  Steps  Take  Place 
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While  In  aost  ways  ^1s  lepleaentatlon  phase  will  be  carried  out  as 
close  as  possible  to  the  way  it  Is  intended  to  be  carried  out  after  the 
progr^  Is  oi^letely  developed,  there  are  seme  constraints.  The 
necessary  Internal  evaluation  of  the  program  will  add  so^  elements 
that  will  (K^t  be  required  later.  Part  of  your  responsibility  here  Is 
to  assure  that  these  a<k^d  factors  modify  the  Instructional  event  for 
the  trainee  to  the  least  |x>ss1ble  ctegree  consistent  with  the  requirements 
of  the  evaluation  plan. 

The  steps  In  implementing  the  management  plan  are  shown  In  the  flow 
chart  In  Figure  IV. 2.  the  fuld-out  page  at  the  end  of  this  block.. 


2.0  PROCEIXJRES 


2.1  Sioplewent  Instructor's  Manual 

One  of  the  critical  Iteas  you  should  have 
received  from  the  team  that  accoopllshed  the 
development  work  in  Phase  III  Is  an  Instructor's 
manual  that  describes  the  course  and  gives 
directions  for  aoninlstering  the  course.  You 
should  thoroi^hly  review  this  dociaisnt.  Ms  a 
mlninjm,  it  should  contain  the  following; 

1.  A clear,  complete  (Ascription  of  the  course. 

2.  A description  of  the  target  population. 

3.  Directions  for  adninistering  and  scoring  tests. 

4.  Directions  for  acini nis ter ing  the  course. 


ur 
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For  more  details  of  what  should  be  included  in  the  instructor's  manual, 
refer  to  Block  I II. 4,  DeVELOP  INSTfOJCTION. 


J-5 

Be  sure  the  Infometlon  you  need  Is  1nc1u(kid  In  the  aanual.  If  you 
ere  not  satisfied  with  the  aanual,  get  back  with  those  who  developed  It 
and  attempt  to  resolve  differences  or  obtain  the  missing  Infomitlon.  If 
you  still  (k>  not  have  adequate  Information  to  carry  out  your  responsibilities, 
do  what  the  manual  says-^even  If  you  disagree.  But  docunent  what  yog  think 
should  have  been  different;  the  ISO  proems  Is  never  co^)1ete.  so 
there  Is  still  time  for  chants  In  any  or  all  parts  of  the  program. 

In  addition  to  a thoi*ough  review  of  the  Instructor's  manual,  you  may 
need  supplemental^’  Information.  The  reasons  for  this  are: 

1.  Unless  the  Instructor's  manual  was  prepared  wlth  your  par- 
ticular training  facility  In  mind,  some  necessary  details 
may  be  missing.  You  m^y  have  to  provicte  details  of  how  the 
management  plan  will  be  Integrated  with  the  rules  and  regu- 
lations of  your  particular  comnand,  and  witn  your  facilities, 
equipment,  personnel,  etc.  You  will  need  to  make  sure  of 
such  essentials  as  scheduling,  equipment  locations,  personnel 
assignments,  contingency  plans,  and  any  other  details  peculiar 
to  your  particular  situation. 

2.  The  instructor's  manual  will  not  include  che  Internal  evaluation 
plan  from  Block  V.l  since  the  Information  is  not  Intended  for 
later  field  use.  The  Instructors  may  be  required  to  coitplete 
forms  or  perform  other  activities  net  included  in  the  manual. 
Observers  and  monitoring  equipment  may  be  present  as  a part  of 
the  evaluation  plan.  The  instructors  must  be  informed  of  all 
such  factors  that  will  have  an  inpact  on  the  manner  in  which 
they  carry  out  the  instructional  function. 


6 


2.2  Suppleirent  Stude.it's  Manuals 
Most  of  what  has  been  said  about 
instructor's  m'nuals  is  also  true  of 
student's  manuals.  The  trainee  must  nave 
a clear  idea  of  what  lie  is  supposed  to  be 
doing  if  he  is  to  make  optimum  use  of  the 
learning  materials.  Again,  you  will  want 
to  thoroughly  re,  Jew  the  student  manui’  and  go  th'^ugh  the  proper  channels 
to  clarify  or  modify  any  unacceptable  areas.  And  again,  as  you  did  for  the 
instructor's  manual,  you  may  need  to  provide  supplementary  materials  des- 
cribing unique  characteristics  of  the  particular  instructional  facility  in 
which  the  trainee  will  participate. 

It  is  particularly  important  to  let  the  trainee  know  the  role  he  is 
p1e(ying  in  this  ISO  effort.  The  particular  course  being  conducted  here 
is  still  in  the  development  stage,  and  the  trainee  can  help  make  it  a 
better  program.  You  need  him  and  hi;  cooperation.  If  you  tell  him  this, 
he  is  likely  to  respond  in  a positive  way.  Most  likely,  the  evaluation 
plan  will  require  nunrerous  inputs  from  the  trainee  that  art  not  1’’  ted 
in  the  student's  manual.  You  must  supplement  the  manual  by  providing  the 
trainee  with  clear  instructions,  not  only  as  to  what  is  expected  of  him, 
but  also  how  the  information  will  be  used.  This  instruction  may  be 
developed  for  the  trainees  in  any  suitable  manner  including  an  oral  presen- 
tation. 
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2_.2_Tr£in_S^f 

The  instructor  is  a vital  part  of 
any  instnjcticnal  system,  and  the  effort 
put  into  helping  prepare  him  to  undertake 
the  functions  expected  of  him  will  be 
effort  well  spent.  Hany  Instructors  are 
not  acciBtomed  to  Instruction  that 
emphasizes  how  the  student  perfoms  rather 
than  how  the  instructor  perfome.  However, 
the  instructor  is  just  as  critical  to  the 
ISO  team  as  in  a traditional  setting.  A 

dlear  understanding  on  his  part  of  the  critical  functions  that  he  must 
perform  will  help  him  fit  into  his  role.  Since  the  instructor  will  con- 
trHbute  heavily  to  the  planning  and  carrying  out  of  the  implementation 
phtse,  he  should  be  brought  into  the  team  as  soon  as  practical. 

The  Instructor  often  will  be  scwwone  other  than  the  designer,  developer, 
or  evaluator  of  instruction.  He  must  be  trained  for  his  role  as  an  instruc- 
tor, and  must  be  able  to  demonstrate  his  ability  to  work  effectively  in  the 
particular  training  setting.  An  example  of  one  instructor  evaluation  pro- 
gram follows; 


EXAMPLE 


Instructor  Training  Objectives 
ACTION;  Conduct  training  that  causes  learning. 

CONDITIONS;  The  student  will  be  given; 

1.  A Trainer  Appraisal  Kit  (TAK) 

2.  A group  of  four  to  six  students 

3.  A requirement  to  arrange  for  problem  support  and  a problem 
site 
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4.  Preparation  time: 

a.  14  duty  hours  for  a student's  first  presentation. 

b.  8 duty  hours  for  subsequent  presentations  (except 
for  failures) 

c.  4 duty  hours  on  all  repeats  of  failed  TAK's, 

5.  Media  Assistance  Office  resources 

STANDARDS; 

1.  802  of  the  students  will  meet  the  learning  standard  of 
ti»e  TAK. 

2.  Presentation  time  will  not  exceed  the  time  limit  specified 
in  paragraph  4b  of  the  TAK. 

The  instructor  trainee  delivers  the  instruction  and  pre-  post-tests  to 
the  students,  and  evaluates  his  own  performance  in  terms  of  t^e  amount  the 
stuftents  learned. 

He  is  also  evaluated  in  tenris  of: 

1.  classroom  management 

2.  instructor  qualities 

3.  control  of  interest 

4.  lesson  organization 

5.  establishing  a good  learning  environment 

6.  improvising  training  aids 

The  itemized  checklists  and  the  scoring  procedures  and  criteria  are 
shown  in  Appendix  A,  page  ig. 

The  instructors  must  become  thoroughly  familiar  with  the  particular 
course  Make  sure  they  have  thoroughly  reviewed  the  instructor's  manual, 
student's  manual,  supplementary  instructions,  and  all  of  -t-he  learning 
materials  and  equipment  with  which  they  will  be  involved.  In  some  cases, 
specialized  training  will  be  required.  For  example,  if  they  are  to  teach 
students  to  operate  a certain  piece  of  equipment,  they  may  have  to  fi^st 
learn  to  operate  it  themselves. 
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The  Instructors  must  be  Informed  of  the  techniques  and  purposes  of 
the  internal  evaluation  plan  that  will  be  carried  out  during  the  course. 

Since  they  probably  will  be  involved  both  as  evaluators  and  as  one  of  the 
variables  being  evaluated,  they  should  have  the  opportunity  to  review 
what  is  being  evaluated  and  why. 

Most  likely  the  instructors  will  be  responsible  for  the  testing  of 
students;  therefore,  they  must  be  thoroughly  familiar  with  how  the  tests 
are  supposed  to  be  administered.  When  performance  tests  are  to  be  used, 
special  Instructor  training  In  test  administration  may  be  required. 

A Trainer  Appraisal  Kit  (TAK)  as  used  In  tho  Trainer  Development 
Program  (TRADEP)  determines  if  the  student  can  prepare  and  present  instruc- 
tion In  such  a manner  as  to  cause  learning.  Each  student,  upon  being 
assigned  a TAK,  receives  a folder  containing  administrative  Instructions, 
a training  objective,  a sample  test  Item,  and  background  material  on  the 
subject  to  be  presented.  The  TAK  subject  will  be  new  to  the  majority  of 
TRADEP  students,  yet  will  be  one  which  <*an  be  effectively  taught  in  15-25 
minutes  (a  specific  time  limit  is  designated  in  each  TAK).  After  receiving 
the  TAK,  the  student  studies  the  training  objective  and  background  material, 
selects  a method  of  instruction,  designs  and  rehearses  his  class,  then 
presents  it  to  a group  of  4-6  students.  The  preparation  time  allowed  for 
each  TAK  varies  according  to  tne  number  of  TAK's  the  student  has  already 
presented,  but  will  always  be  specifically  defined  by  the  team  monitor. 

In  lAK  presentations,  causing  learning  in  students  and  instructor  presen- 
tation skills,  are  emphasized  co-equally.  Otiter  instructional  modules, 
previously  studied,  are  also  tested  during  a TAK  presentation.  The  tasks 
of  the  training  objectives  to  be  graded  are:  conduct  training  that  causes 
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laaming;  demonstrate  movement  and  gesturing  techniques;  prepare  a 50. 

100.  and  200>man  classroom  for  Instruction;  establish,  maintain,  akwl 
1ny>rove  a good  learning  situation;  1n^>rov1se  training  alcb;  demonstrate 
the  correct  use  of  the  pointer.  See  Appendix  A for  the  Instructor 
presentation  techniques  checklist. 

Prior  to  the  student's  presentation  of  his  TAK,  the  team  monitor 
Issues  the  students  a pretest  to  screen  out  those  Individuals  who  can 
already  perform  the  training  objective  to  standard  as  well  as  identifies 
those  students  who  do  not  meet  the  entr>'  level  requirements  for  the  TAK. 

The  student  then  presents  his  lesson. 

After  the  TAK  presentation,  the  team  monitor  administers  a posttest 
for  the  TAK  training  objective.  The  results  of  this  posttest  will  be 
used  by  the  student  to  deduce  whether  or  not  the  mlnimisn  learning  standard 
has  been  met,  and  as  a basis  for  writing  a list  of  recommended  changes  to 
alter  the  presentation  of  the  TAK.  Additionally,  the  team  monitor 
critiques  the  stu<tent  on  his  presentation  techniques. 

The  TAK  demonstrates  a logical  method  of  improving  Instruction  using 
evidence  rather  than  opinion,  as  justification  for  change.  The  practice 
the  student  receives  in  the  preparation,  presentation,  and  validation  of 
instruction  will  make  him  a more  effective  instructor  when  he  returns  to 
his  department. 

EXAMPLE 

Trainer  Appraisal  Kit  (TAK) 


1.  TAK#:  MAC  18  (B) 

2.  Characteristics  of  Typical  Learners:  See  your  team  monitor 
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3.  Title:  Use  of  Hilitarv  Explosives 

4.  Tlroe~L1mlts: 

iT  Prepare t1 on  time:  Sec  your  team  monitor, 
b.  Presentation  time:  15-20  ml nutes . 

5.  Ne<assary_Equ1pTOnt  and  Resources:  None.  Self-contained. 

6.  Training  Objectives" 

TSSKl  Matcn  each  typo  of  military  explosive  with  Its  principal  ise. 
CONDITION:  Given  a list  of  12  military  explosives  and  a list  of 
ten  uses. 

TRAINING  STANDARD:  Ten  of  twelve  answers  must  agree  with  Table  1 of 
the  Background  Information,  para  8,  below. 

7.  Typical  Posttest~Iteiin(sl  or  Performance  ~^est:  Here  Is  a list  of  common 
military  armored  vehlues  and  a list  of  the  principal  uses  of  comnon 
military  armored  vehicles.  Write  the  lettor  corresponding  to  the  name 
of  each  vehicle  behind  the  number  on  the  answer  sheet  which  corresponcte 
to  Its  principal  use. 


A. 

Simple  Item: 
H114 

1. 

Troop  Transport 

8. 

M113 

2. 

Tank  Retrieving 

C. 

M88 

3. 

Reconnaissance 

D. 

M578 

NOTE:  Some  principal  uses  may  be  matched  with  more  than  one  vehicle. 

8.  Background  Information: 

a.  Military  Demolitions.  Military  demolitions  are  the  destruction  by 
fire,  water,  explosive,  mechanical,  or  other  means,  of  areas,  struc- 
tures, facilities,  or  materials  to  accc-iplish  the  military  objective. 
They  have  offensive  uses,  for  example,  the  removal  of  enemy  barriers 
to  facilitate  the  advance  and  the  construction  of  friendly  barriers 
to  delay  or  restrict  eneny  movement. 

b.  Dt:f1n1t1ons. 

TiT^xplosfves.  Explosives  are  substances  that,  through  chemical 
reaction,  violently  change  and  release  pressure  and  heat  equally 
in  all  directions.  Explosives  are  classified  as  lov.  or  high 
according  to  the  detonating  velocity  or  speed  (feet  per  second) 
with  which  this  change  takes  place. 

(2)  Lov!  Explosive.  Low  explosives  deflagrate  or  change  from  a solid 
to  a gaseous  state  relatively  slowly  over  a sustained  period. 

This  quality  makes  the  low  explosive  id^al  for  pushing  or  shoving 
a target.  Examples  are  the  smokeless  and  black  powders. 

(3)  High  Explosives.  The  change  In  this  type  of  explosive  to  a 
gaseous  state— detonation— occurs  almost  Instantaneously,  producing 
a shattering  effect  upon  the  target.  Detonation  retes  range  from 
1,000  meters  [>er  second  (3,280  feet)  to  8,500  meters  per  second 
(27,888  feet).  High  explosives  are  used  where  this  shattering 
effect  is  requird— in  certain  demolition  charges  and  in  charges  in 
mines,  shells,  and  bonijs. 


flfJJIllfflfflipmrimiijim 
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TMT 


Tetrytol 


lack 

owder 


Characttri sties  of  Explosives 


PRINCIPAL  USE  APPROX.  VELOCITY 
OF  DETONATION 


Main  charge* 
booster  charge, 
cutting  and 
breaching 
charge,  general 
and  nllltarv  use 
in  forvard 
areas 


Catering  and 
ditching 


Tine  blasting 
fuze 


feet/sec) 


RELATIVE  PACKAGING 
EFFECTIVE- 
NESS AS 
EXTERNAL 
CHARGE 
(TNT- 1. 00) 


26,379 

fps 

3,400 

Rips 

11,000 

fps 

400 

mps 

1,312 

fps 

TAK  MAC  18  (B) 

Posttest;  Write  the  letter(s)  corresponding  to  the  name  of  each  type 
of  iTTTtary  explosive  following  the  number  on  the  answer  sheet  which 
corresponds  to  its  principal  use. 


A.  Amatol  80/20 
8.  RDX 
C.  PETN 

0.  Straight  Dynamite  {Coawercial) 

E.  Anonlun  Nitrate-Notranan 

F.  Tetrytol 


1.  Catering  and  Ditching 

2.  Shaped  Charges 

3.  Detonating  Cord,  M118-Blast1ng  Caps 

4.  Quarry  and  Rock  Cuts 
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2.4  Monitor  Stu<tent  Selection 

The  Instructor's  manual  will  have  def1r.ed 
the  population  for  which  the  course  was  designed. 

That  Is,  the  required  entry  behaviors  of  students 
will  have  been  outlined,  and  probably  tests  or 
other  appraisal  devices  Included  to  help 
determine  If  the  prospective  students  are  under- 
qualified  or  over-qualified  to  take  the  course. 

Remedial  steps  might  be  Included  as  a part  of  the 
learning  package  to  help  under-qualified  students 
learn  the  requ1(*ed  skills  and  krxiwledge  before 
beginning  the  major  portion  of  the  course.  In  any  case,  determine 
whether  the  students  meet  the  target  audience  specifications.  This 
Information  will  be  a useful  input  to  the  Internal  evaluation  program 
to  be  discussed  In  Block  V.l. 

The  nuid>er  of  students  selected  may  be  determined  by  agreement  between 
those  who  developed  the  course  and  those  who  will  evaluate  the  course, 
and  the  training  facility,  personnel,  etc.,  constraints,  or  the  number 
may  be  <teterw1ned  by  normal  course  flow.  In  either  case,  find  out  how 
many  students  are  to  be  involved. 


2.S  Secure  Time  Allocation,  Space,  Materials 
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2.5.1  Secure  Time  Alldcatlon 

1.  Determine  when  Instruction  will  begin  and  end.  The  overall 
course  time  will  be  Indicated  In  the  Instructor's  manual. 
However,  this  often  Is  Just  an  estimate.  Arrange  for  extra 
time  If  possible,  even  If  It's  after  regular  hours.  Also, 
decide  wliat  to  do  with  the  students  1f  they  finish  earlier 
than  planned. 

2.  Time  allocations  per  topic  or  block  of  Instruction  should 
be  listed  In  the  instructor's  manual.  These  times  may 
determine  when  and  for  how  long  certain  equipment  and 
facilities  need  to  be  reserved.  However,  again  keep  In 
mind  that  these  times  generally  are  estimates. 

3.  In  self-paced  programs,  some  individuals  will  finish  before 
others.  Self-pacing  will,  however,  be  meaningless  If  the 
faster  learners  have  nothing  to  do  after  they  finish  their 
work.  Sometimes,  faster  learners  are  permitted  to  leave 
and  go  to  their  next  duty.  Sometimes  they  can  be  utilized 
as  peer  tutors  to  help  slower  learners.  Check  the  manage- 
ment plan  and  request  modification  of  it  within  local  con- 
straints If  necessary. 

2.5.2  Secure  A<tequate  Space 

1.  Make  sure  the  allocated  space  is  adequate  for  the  number  of 
students.  Instructors,  and  evaluation  personnel,  the  kinds  of 
Instructional  activities  Involved,  and  the  nunter  and  kinds 
of  Instructional  equipment. 
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2.  Nike  sure  space  Is  adequately  heated,  cooled,  lighted, 
noise  controlled,  and  has  adequate  safety  precautions. 


2.5.3  Secure  Adeouate  Haterlals 

1.  Check  the  Instructional  materials  to  nake  certain  the 
oaterlals  are  what  they  arc  supposed  to  be  and  that  you 
have  sufficient  copies  for  all  *:he  stiwlents. 

2.  Make  sure  all  evaluation  materials  are  iivallable. 

3.  Check  audio  and  video  tapes.  Hake  sure  the  correct  items 
are  In  the  correct  quantities,  and  that  Items  are  clearly 
Identified. 


2.5.4  Secure  Adequate  Equipment 


1.  Make  sure  all  equipment  Is  available  and  operable. 

2.  Make  sure  operators  will  be  available,  and  who  to  contact 
In  case  of  equipment  malfunction. 


2.5.5  Miscellaneous 

Even  If  ycu  have  done  a perfect  job  of  planning,  you  can  depend  on 
something  going  wrong.  Do  not  set  up  a tight  schedule  for  yourself 
for  the  periods  of  time  when  Instruction  Is  taking  place.  Probably 
you  will  be  busy  with  contingency  plans  to  work  your  way  around  the 
unexpected.  Be  sure  you  have  access  to  your  supervisor  In  case  of 
insurmountable  problems. 
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3.0  OUTP'JTS 

The  outputs  of  this  block  should  coieist  of; 

3.1  Products 

1.  Supplenentary  Irstructlons  given  to  Instructors  (see  Example 
page  17  ). 

2.  Supplementary  Instructions  given  to  students  (see  Exao^le 
page  17). 

3.2  Other  Dociwentatlon 

1.  Outline  of  special  training  given  to  Instructors  or  other 
staff  members. 

2.  Ntxtiber  of  students  Included.  Rationale  for  their  selection. 

Any  pertinent  Information  about  student  qualifications  and 
any  r^equlred  remedial  work. 

3.  Outline  of  time  allocation  plan,  facility,  and  equipment  plan, 
and  any  additional  student  management  plans  not  Included  above. 


'iiiiNHii:i!mi;lTi:iiiHn:ifi1TW»rim  
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EXA»^L£S 

Sa^le  s«4)p1enent  to  Instructors  ntanual  for  course:  OH-58 

Hellrapter  Repair 

A.  Course  Description 

1.  Installation  facilities  are  actuate  to  train  up  to  tcer 
support  level  maintenance. 

2.  Two  decomissioned  (M-58  Helicopters  will  be  used  in  place 
of  the  main  transmission  trailer  and  transmission  oil 
system  tralnei*. 

3.  Unit  clerical  su^aort  maintains  stuctent  files. 

4.  Manda  ory  briefings  by  the  school  coomandant  and  base 
connanders ' staff  will  require  one  half  day  during  the 
first  week  of  instruction. 


Sample  supplement  to  student's  manual  for  course;  OH-58 

Helicopter  Repair 

1.  A pretest  will  be  a<hi1n1stered  before  each  block  of 

instruction.  Those  idio  pass  the  pretest  (achieve  901i) 
will  be  from  that  block  and  will  begin  the  next 

Mock  of  Instruction. 

2.  Students  who  fail  the  posttest  at  the  end  of  apy  block 
of  instruction  will  receive  remedial  training.  Those 

who  fall  a block  after  remedial  training  will  be  counseled 
by  the  instructor.  Ihe  Instructor  will  recoinnend  either 
further  remedial  work  or  transfer  from  the  training  course. 

3.  A stuctent- faculty  review  panel  will  consider  all  coipplaints 
between  students  and  faculty. 
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TEST  STANDARD  FOR  CHECKLIST  5.81 


DIRECTIONS:  This  critique  sheet  will  be  filled  out  In  the  following  manner: 

1.  Mark  all  portions  of  each  Item  with  a checkmark  fo:'  pass;  and  an  X for 
fall;  or  an  0 If  not  applicable. 

2.  After  all  portions  of  each  It«n  have  been  Judged,  the  rater  will  r«n- 
plle  the  results  of  each. 

a.  If  all  Dortlons  of  an  Item  are  passed.  It  will  be  marked  with  a 
checkmark  (pass;. 

b.  If  one  or  more  portions  of  the  Item  are  failed,  the  Item  will  be 
marked  with  an  X (fail). 

c.  Non>appl1cab1e  (0)  portions  of  the  Item  will  be  counted  neither  as 
pass  or  fail.  If  all  portions  of  the  Item  are  marked  0,  the  Item  will  be 
marked  0 (not  applicable). 

3.  After  all  Items  have  been  Judged,  the  rater  will  compile  the  results 
of  each  Category. 

a.  If  50X  or  more  of  the  applicable  Items  in  a Category  are  passed,  the 
Category  will  receive  a checkmark  (pass). 

b.  If  less  than  50*  of  the  applicable  Items  in  a Category  are  passed, 
the  Category  will  receive  an  X (faM). 

c.  If  a21  Items  in  a Category  arc  not  applicable,  the  Category  will  be 
marked  0 (not  applicable). 

4.  Training  Objective  5.81  (for  TAK-s)  or  5.82  (for  50-minute  Final)  is 
passed  if  all  applicable  Categories  and  80%  or  more  of  all  applicable  Items 
are  passed. 
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THE  INSTRUCTOR  PRESENTATION  TECHNIQUES  CHECKLIST  (5.81) 

ProbiM  Number  t Title  . _ . 

Date 

(tomlnee 


Category!:  Cl assromn  Management 


Itan  A:  Roan  Appearance  (USAIS  Staff  & Faculty  SOP,  para  1.13 

A para  9.19). 

The  clascroom  Is  clean. 

Desks  are  arranged  neatly. 

It«n  B:  Class-in-Sess1on  lights  are  turned  on.  (Classroom  manage- 
ment Handbook,  p.  6). 

Item  C:  Podium  Cards  (USAIS  Staff  & Faculty  SOP,  para  2.12). 

All  sets  of  podium  cards  are  arranged  with  lesson  title  on  top, 

student  class  in  the  middle,  and  Instructor's  name  on  the  bottom. 

The  correct  nimiber  of  sets  of  podium  cards  \tm  sets  for  50-man 

classroom,  three  sets  for  80-series  classroom,  and  three  sets 
for  a 200-man  classroom)  is  displayed. 

Item  D:  Visitor's  Folders.  (USAIS  Staff  A Faculty  SOP,  para  2.18). 

The  proper  inmiber  is  use..!  {tw  in  a 200-man  classroom,  one  in  all 

others).  (USAIS  Staff  & Faculty  SOP,  pare  2.18a). 

Folder(s)  is  (are)  in  the  correct  location  (one  at  the  table  in 

the  rear  of  the  classrooin  and  for  200-man  classrooms,  one  in  the 
visitor's  booth).  (USAIS  Staff  & Faculty  SOP,  para  ?.18a). 

A pad  of  paper  and  pencil  are  provided  with  each  folder. 

__  Item  E:  Lighting.  (TRADEP  Handbook,  para  6-7). 

Lights  are  turned  up  when  the  instructor  is  the  focus  of  attention 

or  the  students  are  working  on  a pracxical  exercise. 

Lights  are  turned  down  when  modified  lighting  is  necessary  to 

highlight  a film,  slide,  or  some  other  training  aid. 

Iten  F;  Spotlights.  (TRADEP  Handbook,  para  6-7). 

Spotlights  are  used  to  highlight  the  instructor  or  a training  aid. 

Are  they  used  so  as  to  not  "wash  out"  any  visuals? 


item  G:  Curtains.  (FM  2-U.,  Test  Edition,  p-  86)- 


They  are  kept  closed  except  when  opened  for  a video  presen- 
tation or  tc  expose  f training  aid 

When  opened,  only  the  area  necessary  is  exposed  i.e.j  only  i;he 

blackboard  or  screen  it)  use. 

Itar)  H:  Sound.  (Classroom  Hanagement  Handbook,  p.  12). 

The  sound  is  adju,sted  so  that  all  personnel  in  the  classroom 

can  hear  the  presentation  clearly. 

The  sound  level  is  adjusted  so  that  there  1s  no  "feedback" 

(high  pitched  squeak)  in  the  system. 

Item  I:  Special  effects.  Special  effects  are  used  to  add  emphasis 
to  a key  point  or  retain  attention  without  creating  a 
"dog  and  pony  show"  atmosphere.  (TRADEp  Handbook,  para  6-4). 


Ranarks : 


Total  applicable  Items:  

Items  passed:  

Items  passed = % 

Total  applicTole  Itens 

Category  II.  Instructor  Qualities 

Item  A:  Appearance. 

The  instructor's  appearance  is  in  compliance  with  AR  670-5  and 

AR  672-4-1. 

His  appearance  is  neat  (shoes  and  brass  shined,  clean  uniform,  etc.) 

His  posture  is  good.  (TRADER  Handbook,  para  7-18). 


The  uniform  fits  properly. 

Iten  B;  Voice. 

He  can  be  heard  throughout  the  classroom  without  shouting. 

(FM  21-6,  Test  Edition,  p.  85). 

He  varies  the  volume  of  his  voice.  (TRADEP  Handbook,  para  7-14). 

He  clearly  enunciates  (speaks  clearly).  (TRADEP  Handbook,  para 

7-6;  FM  ?-16,  Test  Edition,  p.  85). 

Ha  correctly  pronounces  his  wrds.  (TRADEP  Handbook,  para  7-6). 

He  uses  correct  grammar.  (TRADEP  Handbook,  para  7-7). 

His  rate  of  speech  is  varied.  (IHADEP  Handbook,  para  7-8). 

The  narrative  Is  Interspersed  with  pauses  that  highlight  key 

points.  (TRADEP  Handbook,  para  7-10). 

_ His  voice  projects  sincerity  and  a feeling  of  interest  in  each 

student.  (TRADER  Handbook,  para  7-14;  FM  21-6,  Test  Edition,  p.  84). 

Item  C:  Movement.  (TRADEP  Handbook,  para  7-19). 

His  movement  covers  tte  entire  stage. 

He  uses  “V's"  and  "W's"  to  keep  the  microphone  cord  out  of  his 

way  and  maintain  contact  with  the  entire  class. 

He  moves  naturally  and  with  a purpose. 

He  avoids  blocking  the  view  of  his  training  aid. 

Item  D:  Gestures.  (TRADER  Handbook,  para  7-2C). 

His  gestures  emphasize  the  spoken  word. 

The  gestures  are  visible  to  the  entire  student  body. 

Gestures  are  made  in  a natu' il  manner. 

Distracting  or  antagonistic  gestujes  and  mannerisms  are  eliminated. 
(FM  21-6,  lest  Edition,  p.  84). 

Item  E:  Di.'ectrass.  ;FM  21-6,  Test  Edition,  p.  87). 

tie  maintains  eye  contact  with  the  student  body. 

He  remains  facing  the  student  body,  even  when  moving. 
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Itm  F:  Questioning  Techniques. 

All  questions  asked  mphasize  a point,  keep  students  alert, 

ch«;k  understanding  of  a key  point,  review  material , or  stimu- 
late thought.  (FM  21-6,  Test  Edition,  p.  85). 

All  questions  require  a response  otter  than  "Yes"  or  “No." 

(FM  21-6,  Test  Edition,  p.  85). 

The  "ask  - pause  - call"  technique  Is  used  In  all  questions. 

(FM  21-6,  Test  Edition,  p.  85-86). 

All  questions  are  properly  phrased?  (FM  21-6,  Test  Edition, 

p.  85-86). 

Are  all  student  answers  evaluated  as  right  or  wrong?  (FM  21-6, 

Test  Edition,  p.  86). 

The  question  answering  technique  encourages  further  student  re- 
sponse. (FM  21-6,  Test  Edition,  p.  86). 

Item  F:  Subject  Preparation. 

He  teaches  without  fumbling  for  words  and  showing  uncertainty 

before  proceeding.  (FM  21-6,  Test  Edition,  p.  84). 

The  lesson  is  presented  without  frequent  referral  to  notes  or 

slides.  (TRADER  Handbook,  para  6-7c). 

Remarks : 


Total  applicable  Items: 
Items  passed: 


Items  passed = % 

Total  applicable  Items 

_ Category  III.  Control  of  Interest. 


Item  A:  Student  Involverrent.  The  students  are  Involved  in  the 

teaching  process  through  maximum  use  of  questions  and  practical  ex- 
ercises. (FM  21-6,  Test  Edition,  pp.  6-7  & p.  83).  (TRADER  Handbook, 
pp.  1-3  to  1-9) . 
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It»i  B:  Humor.  Jfcraor  used  in  the  presentation  is  appropriate  (no 

religious,  racial,  or  sexual  overtones).  (FH  21-6,  Test  Edition,  p.  85). 

Itan  C:  Training  Aids.  Training  aids  are  used  where  the  subject  re- 
quires visual  or  audio  support.  (FM  21-6,  Test  Edition,  p.  87-88). 

Item  D:  Interest  Factors.  Interest  factors  are  used  to  contribute 

to  tte  class.  (TRADER  Handbook,  para  8-3). 

Category  IV:  Lesson  Organization. 

Item  A:  Introduction.  (TRADER  Handbook,  para  4-4). 

A Gain  Attention  ;’('p  Is  used. 

The  subject  is  tied  in  to  previous  and  subsequent  instruction. 

The  training  objective  is  fully  explained.  (FM  21-6,  Test 

Edition,  p.  40-42). 

Item  Bj  Body. 

The  class  Is  broken  into  meaningful  segments.  (FM  21-6,  Test 

Edition,  pp.  11-18). 

The  segments  of  the  class  are  presented  in  a logical  sequence. 

(FH  21-6,  Test  Edition,  pp.  18-19).. 

Transitions  are  used  to  move  from  one  segment  to  another. 

■ (TRADER  Handbook,  para  4-9). 

Item  C:  Conclusion.  (TRADER  Handbook,  para  4-5). 

The  students  are  alerted  for  the  review. 

All  main  points  are  reviewed  in  a logical  sequence. 

There  is  a strong  concluding  statement. 


Remarks : 
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Total  applicable  Items: 
Itmns  passed: 


ItCTs  passed = % 

Total  applicable  Items 

CateQory  V.  Establishes  a Good  Learning  Environment. 


I ten  A:  The  nominee  coranunicated  an  OPTIMISTIC,  HIGH  STANDARD  OF 
EXPECTATION  to  the  group. 

Item  B:  The  objective  was  NEGOTIATED  (EXPL  until  all  the 

group  came  to  see  it  as  valid,  important,  atu  inaole,  and  personally 
valuable. 

Item  C:  The  nominee  provided  at  least  one  INCENTIVE  (.angible  or 

intangible)  for  good  performance. 

Item  0:  The  nominee  “REINFORCED"  any  (and  all)  approximaliott.'  of 

behavidr(s)  in  the  objective(s)  occurring  during  the  practice  por- 
tion of  the  lesson  accomplished. 

, Item  E:  The  nominee  "EXTINGUISHED"  any  (and  all)  inappropriate 
actions  of  the  group. 


Remarks : 


Total  applicable  Items: 
Items  passed: 


Items  passed  - % 


Total  applicable  Items 
Category  VI:  Improvises  Training  Aids. 


tern  A:  The  training  aid(s)  used: 
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Reinforced  the  spoken  word. 

Directed  learner  thinking  to  specific  1ten(s). 

Aided  the  learner  to  perform  the  objectives. 

Made  things  clearer. 

Aided  In  retention. 

Iten  6:  The  training  aids  were: 

Appropriate.  Simple,  Accurate,  Necessary,  Attractive. 

Not  a distraction. 

Large  enough  to  be  seen  by  everyone  In  the  classroom. 

Not  used  as  a crutch  by  the  Instructor. 

In  the  proper  position  on  the  platform. 

Removed  or  covered  when  no  longer  required 

Item  C:  If  an  Overhead  Projector  (Vu-graph)  was  used: 

The  slides  used  were  neat  and  attractive. 

Slides  did  not  contain  too  much  material. 

The  lettering  was  large  enough  to  be  seen  by  all  learners. 

The  projector  was  turned  off  when  no  longer  required. 

Item  D:  If  the  chalkboard  was  used; 

The  lettering  was  of  proper  size  to  be  seen  by  all  learners. 

The  lettering  was  neat. 

Learners  were  asked  to  assist  in  developing  material. 

Too  many  abbreviations  were  used. 
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Total  applicable  Items: 

I tans  passed: 

Items  passed 

Total  applicable  Items 


SUWATION 

Applicable  Categories  passed 
Applicable  Itans  passed 
% Applicable  Categories  passed 
% Applicable  Items  passed 
60/ No-Go 


Applicable  Categories  failed 
Applicable  Itans  failed 


INSmUCTIONAL  MATERIALS 
AND  MANAGEMENT/OELIVERY 
SYSTEM  PLAN  FROM  “HASEIQ. 
EVALUATION  PLAN 
FROM  BLOCK'S;. 1 
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MANUAL 

2.1 
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STUDENT'S 

MANUAL 
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TRAIN  STAFF 
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SECURE  TIME  ALLOCATION, 
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FIGURE1S;.2;  Flowchart  of  Block ISL.I:  IMPLEMENT  INSTRUCTIONAL  MANAGEMENT  PLAN 


BLOCK  IV. 2:  CWIDUCT  INSTRUCTION 
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OVERVIEW 

After  the  instructor  has  been  oriented  to  the  instructional  materials, 
and  has  mastered  the  necessary  skills,  he  conducts  Instruction  in  accord- 
ance with  the  instructor's  manual.  As  a part  of  the  instructional 
activity,  he  documents  any  required  changes  and  other  observations,  and 
participates  in  follow-up  activities. 
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CONDUCT  INSTRUCTION 
1.0  INTROUUaiON 

From  a prdctical  point  of  v1a<.  It  Is  almost  Impossible  to  design 
Instruction  that  will  provide  solutions  to  all  possible  problens.  Swte 
students  will  always  have  trouble  mastering  certain  objectives,  and 
performing  certain  tasks.  The  trouble  spot^  will  be  different  for  dif- 
ferent people.  One  of  the  major  duties  of  the  instructor  Is  to  identify 
such  problems  and  to  provide  assistance  where  needed.  There  will  be 
students  who  are  unusually  fast  or  slow.  The  more  capable  students  must 
be  kept  from  becoming  disinterested  and  bored,  and  the  problems  of  the 
less  capable  students  must  be  diagnosed  and  appropriate  action  taken. 

While  the  ISO  approach  can  make  certain  provisions  for  training  individuals 
of  varying  capabilities  and  degrees  of  motivation,  It  continues  to  depend 
on  classroom  managers  or  Instructors  to  meet  unexpected  requirements. 

In  self-paced  instruction,  lectures  are  minimized  and  the  Instructor 
relies  on  prepared  materials  and  other  delivery  systems  to  present  much 
of  tiie  Instruction.  This  frees  the  Instructor  from  many  of  the  routine 
tasks  of  teaching  so  that  he  can  provide  both  academic  and  personal  atten- 
tion to  individual  students.  He  must  diagnose  student  problens  and  plan 
actions  to  remediate  those  problems.  The  process  of  providing  guidance 
or  of  counseling  individual  students  may  be  a new  role  for  some  instructors, 
yet,  it  is  a most  important  function. 
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As  developments  In  the  concepts  of  self-paced  instruction  grw,  the 
Instructor  will  become  increasingly  more  professional  as  a manager  of 
instruction.  He  will  be  able  to  devote  more  of  his  attention  to  the 
problems  and  needs  of  the  Individual  student.  As  such,  he  assumes  dif- 
ferent roles:  that  of  an  FOJT  supervisor,  the  manager  for  a correspondence 
course,  a classroom  manager,  or  an  individual  tutor  and  counselor. 

It  Is  probably  fair  to  say  that  at  this  particular  point  in  the  ISO 
process,  the  emphasis  is  primarily  on  the  instructor  as  a manager  of 
instruction.  This  is  true  because  one  of  the  primary  purposes  of  con- 
ducting instruction  is  t&  insure  that  the  instructional  program  will 
accomplish  what  it  is  intended  to  accomplish.  If  the  program  succeeds, 
it  should  do  so  cn  its  <Mn  merits  and  not  because  of  a unique  performance 
on  toe  part  of  the  Instructor,  for  this  t'eason,  the  primary  responsi- 
bility of  the  instructor  In  this  block  is  to  conduct  instruction  as 
close  as  possible  to  the  method  outlined  in  the  instructor's  manual. 

The  instructor  is  a crucial  member  of  the  ISO  team,  and  as  such  he 
must  be  involved  in  certain  preparatory  steps  prior  to  conducting 
instruction,  and  nwst  participate  in  certain  critical  post-instruction 
acti  vi  ties. 

The  steps  in  the  conduct  instruction  function  are  shown  in  the  flow- 
chart in  Figure  IV. 3,  thy  fold-out  page  at  the  end  of  this  block.  These 
steps  will  now  be  discussed  in  some  detail. 
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2.0  PROCEIXiRE 

2.1  Revlwrt  all  Course  Pocwientatlon 

The  Instructor  should  thoroughly  review 
all  the  course  docimntation  Including  the  Instruc- 
tional materials,  tests,  instructor's  manual, 
student's  manual,  and  evaluation  plan.  Some  of 
these  Items  may  still  be  in  a formative  stage 
of  developnient.  In  this  case,  the  instructor's 
ideas  can  beccmie  an  in^rtant  input  to  completion 
of  the  Items. 

Be  thoroughly  familiar  with  the  instructional  materials.  If  special 
training  is  required  to  provide  the  necessary  expertise  in  some  areas, 
be  sure  you  have  it.  If  you  do  not  fully  understand  all  parts  of  the 
Instructor's  manual,  and  the  additional  Instructions  added  in  Block  IV. 1, 
you  should  make  the  problems  known,  ano  suggest  imprxivements . 

Since  instruction  produced  according  to  ISO  principles  will  have  tests 
associated  with  units  or  modules  of  instruction,  one  of  the  instructional 
duties  is  to  administer  these  tests.  It  is  essential  that  the  test  adminis- 
trator be  totally  familiar  with  the  tests  and  that  he  understands  the  agreed 
upon  standards  for  satisfactory  student  performance.  Many  of  the  tests  will 
be  performance  tests  and  he  will  need  to  be  pirticularly  attentive  to  the 
procedure^  for  a(tai1ni storing  them. 

In  some  situations,  your  only  function  may  be  to  administer  tests. 

This  approach  has  the  advantage  of  consistency— a limited  nianber  of 
specialists  administer  the  tests  to  all  students.  In  other  situations, 
you  may  provide  fhe  instruction  and  adninister  the  tests  to  the  trainees. 
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This  sitiMtion  is  less  (teslfable  because  it  tends  to  cause  a false  sense 
of  c(^)et1t1on  among  instructors.  Also,  it  is  m>t  unconsion  for  instructors 
to  hint  or  advertise  the  fwints  on  which  there  will  be  test  questions,  or, 
in  response  to  student  qi^stions,  present  unusually  detailed  answers  only 
on  those  points  known  to  be  on  the  test.  The  unfortunate  outcome  from 
such  practices  is  to  make  students  appear  to  know  more  than  they  really  do, 
potentially  penalizing  them  when  they  get  to  the  job.  If  you  must  give 
instruction  and  also  administer  tests,  make  sure  to  follow  the  testing 
rules  as  precisely  as  possible  in  order  to  avoid  halo  effects  and  errors 
of  standards. 

Since  most  of  the  tests  will  be  scored  on  absolute  standards,  and  since 
or^  purpose  of  ISD  is  to  get  the  maximum  possible  percentage  of  students  to 
meet  the  criterion,  many  students  will  have  to  be  “recycled"  to  those  parts 
of  the  course,  module,  or  unit  that  they  did  not  pass.  You  must  make 
decisions  In  such  matters  on  the  basis  of  the  testing  rules  agreed  upon 
by  the  whole  ISD  team  and  on  the  student's  performance.  You  will  be  In 
the  position  of  deciding  when  the  student  has  met  all  expectations  of  the 
course  and  is  ready  to  move  on  to  the  next  assignment. 

Rementer  that  when  the  trainee  reports  to  Itis  base,  ship,  or  unit, 
he  is  the  product  of  your  course.  If  he  was  not  properly  instructed,  he 
is  not  likely  to  perform  well.  If  he  was  passed  on  the  test  when  he 
should  not  have  been,  he  is  likely  not  to  be  able  to  perform  on  the  job. 

On  the  other  hand,  if  he  is  required  to  meet  un»^al  or  unreasonable 
standards,  he  may  never  get  out  of  the  course.  For  these  reasons  you 
should  follw  the  testing  instructions  as  closely  and  fairly  as  possible. 
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In  some  self-paced  courses,  students  will  be  In  a hurry  to  finish, 
either  because  they  work  faster  than  other  people,  or  because  they  are 
In  a hurry  to  get  where  they  are  going  next.  Other  students  may  enjoy 
the  benefits  of  the  location  of  *.he  school  and  atten^t  to  make  a career 
of  the  course.  The  instructor  is  in  the  best  position  to  work  with  and 
counsel  students  on  their  course  progress.  Increasing  the  speed  with 
which  each  student  coir|)1etes  the  course,  even  by  a modest  amount,  can 
have  an  iirportar.t  Intact  on  the  cost  of  the  course  and  the  number  of 
trained  people  available  to  the  battalion,  squadron,  or  fleet. 

2.2  Obtain  Required  Training 

You  as  the  instructor  need  to  be  thoroughly 
familiar  with  the  subject  you  will  be  teaching. 

This  may  require  taking  special  training  in 
the  subject-matter  area.  Particularly  when 
teaching  the  use  or  maintenance  of  a new 
weapon  or  piece  of  equipment,  you  will  first 
need  to  become  somewhat  of  an  expert  in  handling 
the  equipment.  Even  if  you  are  an  expert  in 
the  subject-matter  area,  you  still  may  need  training  in  the  use  of  the 
particular  delivery  system  to  b'^  used  in  the  course. 

You  may  need  special  preparation  iii  administering  performance  tests. 
Following  are  steps  you  should  take  when  performance  testing  is  required, 

1.  Read  the  test  several  times,  so  that  you  understand  what 
is  required  by  it. 

2.  Assemble  the  necessary  equipment  for  giving  the  test  and  lay 
it  out  as  stated  in  the  test  conditions. 
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3.  Go  through  each  performance  measure  yourself,  doing  It  as 
required  In  the  test.  Go  through  the  measures  several  times 
until  you  can  perform  them  with  ease. 

4.  Have  another,  qualified  person  give  the  test  to  you,  so  that 
you  can  observe,  as  you  are  being  tested,  test  administration 
by  someone  who  is  skilled  in  giving  the  test. 

5.  Give  the  test  on  a try-out  basis.  This  allows  you  to  practice 
giving  the  test;  that  is,  in  reading  the  instructions  so 

thev  ca.,  '‘nderstood,  ratim  performance,  and  briefing 
participants  in  what  *' ?«  did  correctly  and  incorrectly. 

2.3  Conduct  Instruction  and  Roci/^  i.l  Obsetvatlon 


As  a member  of  the  instructional  staff 
of  a course  produced  by  ISO,  you  have  been 
oriented  to  the  course,  have  mastered  skills 
necessary  to  conduct  the  course,  and  are  ready 
to  start  the  first  group  of  students.  The 
adequacy  of  space,  facilities,  and  materials 
has  been  checked  out  to  make  sure  you  have 
enough  instructional  materials  and  resources  for  the  staff  and  the  students. 
Your  job  is  to  follow  the  instructor's  manual  to  provide  the  ISO  instruction 
for  the  students,  and  collect  data  that  will  help  in  evaluating  and  improving 
the  course. 

Your  major  responsibilities  are; 
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1.  To  carry  out  Instructional  activities  exactly  as  the  instructor's 
manual  specifies  unless  it  is  impossible  to  do  so. 

2.  To  keep  records,  either  in  your  course  guide  or  in  a log  book 
kept  for  that  purpose  (or  in  both),  of  aH.  changes  from  the 
original  course,  whether  these  changes  are  small  or  large, 
temporary  or  lasting,  and 

3.  To  make  specific  notes  on  problems  students  have  with  the 
learning  materials . 

The  reason  for  the  above  Is  that  the  evaluation  of  the  course  is 
supposed  to  be  on  the  course  as  It  is  delivered  to  you  from  Block  IV. 1. 

Any  changes  must  be  considered  in  the  evaluation  so  that  the  instruc- 
tional staff  and  management  personnel  can  make  recommendations  based  on 
what  actually  happened.  For  example,  if  an  audio-visual  lesson  is  so 
consistently  a problem  to  students  that  the  instructor  has  to  replace 
that  particular  lesson  with  tutoring  or  a talk  and  demonstration,  the 
change  from  media  to  instructor  must  be  documented.  If  it  is  not  docu- 
mented, any  test,  results  the  instructor  gets  from  hi?  demonstration 
could  be  interpreted  as  the  results  irom  the  audio-visual  materials. 

Then  any  necessary  iimirovement  in  the  lesson  could  be  overlooked  for 
future  conduct  of  the  course. 

There  are  three  kinds  of  changes  the  instructor  might  have  to  make 
during  course  implementation.  They  are  editorial,  procedural,  »nd  content. 

Editorial  changes  generally  do  not  require  approval.  Many  different 
kinds  of  routine,  mechanical  problems  such  as  misspellings,  typooraphical 
errors,  grammatical  errors,  poor  reproduction  quality  in  printing,  missing 
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pages,,  ami  mlsnin^ered  pages  can  be  changed  quickly  on  a master  copy 
and  on  the  Instructor/student  copies  during  the  course.  These  do 
not  change  the  content  or  procedures  of  the  course,  and  asicte  from 
needing  to  be  recorded  for  future  materials  production,  generally 
require  no  additional  action. 

Any  change  In  directions  to  students  is  a procedural  change,  and 
should  be  recorded  along  with  the  reason  for  the  change.  In  addition. 

1f  It  Is  Intended  to  be  a permanent  change  in  course  procedures,  a 
written  request  for  the  change  should  be  filed  In  compllar.ce  with  local 
regulations.  If,  for  example,  extra  instruction  sessions  an;  found 
necessary  at  certain  points  In  the  course  to  reorient  students,  this 
procedure,  aloPT  with  the  reasons  why  it  was  needed,  should  be 
thoroughly  documented.  This  Is  the  type  of  Input  that  facilitates 
accurate  evaluation  and  requires  revision  of  the  course. 

The  Instructional  materials  may  require  changes,  additions,  or  deletions. 
If  such  c need  anses,  approval  should  be  secured  prior  to  making  the 
change.  At  the  very  least,  the  extent  of  and  reasons  for  the  change 
should  be  thoroughly  documented. 

Docunent  all  important  observations  relative  to  the  instruction.  For 
example,  revised  time  estimated  for  different  blocks  of  Instruction,  equip- 
ment problems,  and  inadequacies  of  space  and  other  f...i11t1es  should  be 
docianented. 

2.4  Participate  1n  Follow-Up  Activities 

As  th.e  ISO  team  expert  cn  conducting  instruction, 
you  will  need  to  participate  in  the  follow-up  activi- 
ties In  which  decisions  must  be  made  to  improve  the 
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effect1vei»ss  of  the  Instructional  program.  While  the  extent  of  your 
participation  will  partially  depend  on  the  procedures  of  your  particular 
command,  your  statsn»nts  of  what  happened  during  the  course  and  your 
reconmendatl ons  for  changes  will  be  an  Inportant  input  to  Phase  V of 
the  ISO  model . 

3.0  OUTPUTS 

The  outputs  of  this  block  should  consist  of  the  following  docunentatlon: 

1.  A record  of  actual  times  required  for  different  parts  of 
the  course. 

2.  Ar\y  deviations  from  the  instructional  plan  and  reasons  for 
the  deviations, 

3.  Specific  notes  on  any  problem  areas  In  the  course, 

4.  Any  suggested  changes  in  the  course  and  rationale  for  the 
changes . 
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The  planning  and  management  functions  essential  to  successful  Im- 
plementation of  a course  is  discussed  in  this  document.  Chapter 
16,  “Operation  and  Evaluation  of  the  Instructional  System,"  presents 
helpful  information  dealing  with  the  changing  roles  of  Instructors 
and  students  and  the  functions  of  ;<ianagement  in  an  Instructional 
system. 

Department  of  the  Army.  Trainer  development  program  (TRADER)  progress 
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A Trainer  Appraisal  Kit  as  used  in  the  Trainer  Development  Program 
determines  If  the  student  can  prepare  and  present  instruction  In 
such  a manner  as  to  cause  learning.  This  report  contains  the  train- 
ing objectives  that  the  TRADER  student  uses,  including  the  TAK 
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Department  of  the  Air  Force.  Instructional  systems  develoi^nent 
(AFM  50-2).  Washington,  D."C. : Headquarters,  United  States  Air 
Force,  Air  Training  Conmand,  December  1970. 


The  trends  toward  implementing  the  concept  of  individualized 
instruction  present  several  pi-oblmns  for  management.  One  of  the 
more  immediate  problems  Is  to  teach  the  instructor  the  functions 
expected  of  him  in  the  system  so  that  he  can  be  ready  to  assume 
his  instructional  duties  and  assist  with  validation.  This  and 
other  management  problems  are  discussed  in  the  context  of  con- 
ducting instruction. 


Department  of  the  Army.  Trainer  development  program  (TRADER)  progress 
report.  Ft.  Benning,  Ga.':  United  States  Army  Infantry  School, 
January  1975. 


This  report  presents  the  latest  information  on  the  development  of 
the  Trainer  Development  Program  (TRADEP).  Composed  of  lists  of 
training  objectives  of  the  TRADEP,  the  training  objectives  are  de- 
signed to  provide  the  TRADER  student  with  an  accurate  description 
of  the  performance  expected  of  him  upon  completion  of  training,  which 
is  aimed  at  determining  whether  or  not  he  can  be  an  effective  in- 
instructor. 
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BLOCK  V.l:  CONDUCT  INTEW^AL  EVALUATION 
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OVERVIEW 

Intenval  evaluation  is  planned  and  conducted  primarily  to  determine 
whether  the  instructional  development  goal  has  been  reached.  Data  are 
collected  not  only  to  assess  student  progress,  but  more  importantly,  to 
improve  the  quality  of  instruction.  The  process  consists  of  collecting 
pertinent  progress  and  process  data,  performance  data,  and  information 
from  students,  instructors,  and  other  informed  personnel;  evaluating 
this  data;  and  making  recommendations. 
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CONDUCT  INTEWiAL  EVALUATI(»< 

1.0  INTTODtJCTlON 

The  primary  purpose  of  Internal  evaluation  is  to  determine  wtether 
the  instructional  development  effort  has  accomplished  what  was  Intended. 
While  this  is  the  global  aim,  the  procedures  are  specific  and  are  con- 
cerned with  the  aspects  of  the  course  that  are  sidaject  to  measurement 
and  informed  Judgment.  One  principal  function,  if  not  the  most  impor- 
tant function,  of  the  internal  evaluation  process  is  to  provide  suffic- 
ient good  data  upon  which  to  base  decisions  about  the  instruction. 

The  quality  of  the  evaluation  is  totally  dependent  upon  the  ability 
to  measure  specific  variables  with  accuracy  and  precision.  One  of  the 
guiding  principles  of  ISO  Is  that  wherever  possible,  decisions  about 
instruction  are  made  on  the  basis  of  specific  data  collected  according 
to  established  standards.  The  intention  is  to  move  from  vague  and 
qualitative  statements  about  effectiveness  to  specific  quantitative 
statements  wherever  possible.  Effective  measurement  can  make  this 
intention  a reality. 

Perhaps  a general  example  of  the  sarre  concern  from  another  field 
would  be  helpful.  In  the  early  days  of  flying,  it  was  most  helpful  to 
pilots  to  know  the  direction  of  the  surface  wind  prior  to  landing.  The 
simple  device  of  the  windsock  provided  a visual  indicator  of  the  wind 
direction  and  a crude  measure  of  the  velocity.  While  it  was  relatively 
easy  to  design  an  instrument  which  would  show  the  direction  of  the  wind. 
Informing  the  pilot  of  the  true  velocity  was  a completely  different 
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matter.  As  tine  passed,  this  Information  eventually  could  be  transmitted 
by  radio  to  the  aircraft,  and  the  pilot  could  make  oetter  decisions. 
Today's  avionics  represent  the  results  of  a long  and  directed  effort  to 
Improve  the  Information  provided  to  the  decision  maker,  with  the  longer 
term  purpose  of  Improving  system  perfonnance  and  safety  through  careful 
research,  measurement,  and  evaluation. 

The  last  fifty  years  have  seen  Improvements  in  the  measurement  and 
evaluation  of  hianan  behavior  and  Instruction  almost  a<;  dramatic  as  In 
the  avionics  exan^le  mentioned  above.  First,  It  was  Important  to  find 
features  or  var1ab1.es  which  could  be  measured  reliably;  that  Is,  where 
repeated  measures  by  the  same  or  different  people  would  arrive  at  highly 
similar  results.  Once  the  measures  are  made  reliable,  it  Is  only 
rj^essary  to  accumulate  sufficient  data  to  find  out  what  Uie  measures 
mean. 

In  the  ISO  process  enough  data  on  students  must  be  collected  so  that, 
through  time.  Instruction  can  be  Improved  based  cn  students'  performance. 
Generations  of  teachers  have  collected  data  on  students.  That  Is,  they 
have  recorded  scores  on  tests  given  In  class  and  have  used  the  scores 
as  the  basis  for  assigning  grades  at  the  end  of  a period.  But  what  makes 
the  ISj  process  new  and  unique  Is  that  the  collected  data  are  examined 
not  only  to  assess  student  progress,  but  also  to  judge  and  Improve  the 
quality  of  the  Instruction.  So,  in  addition  to  the  grades  assigned  to 
stu£?ents,  scores  rww  are  recorded  for  each  segment  of  Instruction  to  sc*e 
how  well  it  performs.  If,  through  tin»,  a large  proportion  of  people 
have  troi&le  with  the  same  segment,  it  Is  reasonable'  to  believe  there 
might  something  wrong  with  the  Instruction.  Thus,  data  collected 
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on  students  is  a necessary*  Input  to  the  ISO  process,  not  Just  a means 
of  deciding  which  student  gets  which  grade. 

Evaluators  are  continuously  looking  for  sources  of  better  data.  It 
might  be  useful  to  distinguish  between  two  kinds  of  data:  hard  and  so't. 
Hard  data  are  more  direct  measures  of  the  variable  of  interest.  In  order 
to  test  whether  someone  knows  how  to  measure  and  record  a patient's  blood 
pressure,  the  alternative  choices  are  ranked  as  follows: 

1.  The  best  (hard)  data  would  be  found  in  the  results  of  a JPH. 

2.  A pencil  and  paper  test  for  the  task  would  be  less  direct,  and 
would  not  be  hard  data. 

3.  The  persons  supervisor  could  be  asked  about  the  Invididual 's 
performance. 

4.  .Asking  the  person  directly  if  he  could  perform  the  task  may 
be  even  less  useful  than  asking  a supervisor. 

In  this  list  of  alternative  choices,  the  data  get  softer  in  moving  fro.ti 
direct  observation  of  the  performance  to  the  recording  of  student's 
opinions . 

Data  obtained  from  attitude  or  opinion  surveys  or  other  rctinq  forms 
mac'  be  the  only  collectable  data.  It  Is  soft  bu*  i*  it  is  all  thc  t is 
available,  it  must  be  used.  However,  the  confidence  In  the  conclusions 
drwn  wiil  be  much  lower.  A mt're  complete  discussion  of  the  pi'Oblens 
encountered  ‘m  the  use  of  rating  data  was  given  in  Block  1.3.  Ideally, 
direct  measurement  should  always  be  used. 

Since  Internal  evaluation  is  concerned  with  the  evaluation  of  the 
ISD  process  in  any  selected  instructional  setting,  the  measures  used 
will  be  the  best  possible  trade  between  the  real  world  conditions  and 
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the  measures  Available  In  the  Instructional  setting.  Can  antiaircraft 
gunners  make  an  acceptable  nuiher  of  hits  uncter  battle  conditions  In 
rou<ih  seas?  Since  testing  under  these  conditions  Is  rarely  possible, 
a measure  imist  be  used  that  will  give  the  best  approximation  of  the  real 
world  performance.  In  many  Defense  Occupational  Specialities  (CKlSs).  che 
measures  of  student  performance  can  be  performance  of  the  actual  task. 

In  other  DOSs  some  measures  may  only  be  simulations  of  the  task.  In 
the  case  of  the  Gunner's  Mate,  the  task  most  likely  will  be  simulated, 
but  In  the  case  of  the  legal  clerk  or  yeoman,  the  actual  task  may  be  used. 

In  addition  to  the  performance  data  collected  on  students  through  the 
aefeni  111  strati  on  of  tests,  descriptive  data  such  as  student  ratings,  opin- 
ions, and  reports  of  critical  Incidents  are  collected  on  the  Instructional 
materials  and  processes.  Instructors  are  c valuable  source  of  data.  They 
can  report  problems  with  any  part  of  the  Instruction,  such  as  difficulties 
students  have  had  with  certain  objectives,  time  spent  in  presenting  the 
Instruction,  and  opinions  about  instructional  materials  and  procedures. 

As  these  data  accumulate,  It  Is  possible  to  make  better  decisions.  For 
example,  if  a nimber  of  people  had  difficulty  with  a certain  unit  objective 
and  If  the  instructors  consistently  reported  that  the  materials  were 
Incorrect  or  confusing,  the  evidence  is  mounting  that  something  is  wrong. 

But,  notice  that  If  the  Instructors  had  judged  the  materials  to  be 
confusing  and  the  students  regularly  passed  the  tests,  then  the  problem 
may  be  with  the  instructors,  not  with  Wie  materials.  This  is  an  example 
of  the  value  of  hard  data  (student  performance)  over  soft  data  (opinions 
of  Instructor;)  in  making  the  entire  ISO  process  work.  It  Is  an  example 
of  replacing  collective  opinion  with  hard  facts.  Sources  and  uses  of 
several  kinds  of  data  are  presented  in  Table  VI. 
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. Time  required  for  comparisons 

exercises  : c.  Make  decisions  on  student 

. Total  time  tc  complete  flow 


TABLE  V,1  - Continued 
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Each  block  In  the  model  has  outputs  <n  the  form  of  products  and 
other  docionentatlon.  These  outputs  can  be  evaluated  If  they  are  Judged 
or  rated  In  a systematic  way.  First,  the  evaluator  will  want  to  know  If 
the  outputs  are  consistent  with  the  Instructions  and  the  requirements  of 
the  local  command.  If  there  are  elements  missing,  or  If  It  appears  that 
the  outputs  were  documented  as  a part  of  a "paper  exercise,"  these  facts 
need  to  be  recorded. 

For  example,  a number  of  years  ago,  a military  coimiand  issued  a 
regulation  requiring  the  systematic  design  of  all  courses.  When  the 
regulation  was  put  into  operation,  two  Important  problems  existed: 

1.  Only  a few  of  the  people  who  received  the  ragulation  knew 
how  to  do  what  was  required,  and 

2.  Schools  were  asked  to  report  within  three  months,  on  the 
number  of  courses  which  were  In  compliance  with  the 
regulation. 

The  results  were  predictable  and  bad.  Subsequent  audit  of  a number  of 
courses  that  had  been  reported  to  be  in  compliance  with  the  regulation 
revealed  that  the  required  process  of  recording  information  on  a particu- 
lar form  had  been  accomplished.  But,  instead  of  getting  the  data  from 
the  field  as  the  regulation  required,  the  data  had  been  gathered  from 
class  notes  on  instructors.  Resources  had  been  used  in  an  unproductive 
way  and  the  desired  results  from  the  regulation  never  materialized.  The 
conclusion  reached  by  many;  The  whole  regulation  is  bed--do  away  with 
it: 

This  iroblem  could  have  been  avoided  if  the  products  and  other  docu- 
nentation  had  been  evaluated  by  an  internal  evaluation  team  and  properly 
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Inspected  by  higher  heackjuarters . The  first  finding  would  have  been  that 
the  reporting  requirements  wre  unrealistic  in  terms  of  time.  It  is 
highly  unlikely  that  the  ISO  procedures  for  a course  could  he  con^>1eted 
In  three  months.  The  second  findings  probably  would  have  been  that  there 
were  Insufficient  trained  personnel  to  execute  the  rtqulrements,  even 
with  adequate  time.  These  two  findings  could  have  allowed  management 
to  take  corrective  action. 

A final  form  of  Internal  evaluation  deals  with  the  progress  and 
schedule  of  the  ISO  effort.  When  It  Is  decided  to  start  new  Instructional 
programs  or  revise  old  ones  using  ISO  procedures,  a project  schedule 
should  be  developed  using  Program  Evaluation  Review  Technique  (PERT)  or 
other  suitable  methods.  The  manager  of  the  ISO  program  should  keep  records 
of  progress  and  problems  so  that  those  affected  can  make  logical  manage- 
nient  decisions.  For  example,  if  the  personnel  system  is  to  assign  stu- 
dents. a reasonable  estimate  of  the  development  and  start-up  time  must 
be  known. 

The  internal  evaluation  group  should  have  tne  ability  to  review  the 
progress  of  the  effort  and  to  make  an  evaluative  report  of  the  progress 
to  management.  Evaluation  is  not  Intended  to  be  a blame-placing  activity. 
It  Is  concerned  with  the  accuracy  of  the  problem  statement,  and  not  who 
is  at  fault.  Another  of  the  functions  Is  to  try  to  avoid  allowing  problems 
and  tasks  to  “fall  between  the  cracks,”  to  identify  problem;  so  the 
nanager  can  assign  responsibility  for  their  correction  to  a specific 
office  or  person. 

The  next  section  will  discuss  the  specific  procedures  f^r  carrying  out 
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the  Internal  evaluation  function.  The  steps  in  conducting  Internal 
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evaluation  are  shown  In  Figure  V.l.  the  fold-out  page  at  the  end  of 
this  block. 


2.0  PIfflCEWRE 

2.1  Develop  Internal  Evaluation  Plan 
0r.ce  a sound  Internal  evaluation  plan 
has  been  developed  using  the  System  Master 
Plan  from  III. 2 as  a basis,  implementation 
of  the  plan  is  a relatively  straightforward 
effort,  and  interpretation  of  the  collected 
data  is  made  easier.  Since  internal  evalua- 
tion begins  very  early  In  the  ISD  model. 


planning  must  beoln  early.  Much  of  ttie 

student  and  Instruction  data  will  be  collected  during  the  CONDUCT 
INSTRUCTION  effort  in  Block  IV. 2.  Those  responsible  for  Implementing 
the  instructional  management  plan  and  conducting  instruction  will  need 
to  know  the  internal  evaluation  plan  before  they  can  carry  out  their 
functions . 

The  internal  evaluation  plan  must  iitrlude  plans  for  collecting' data 
from  a number  of  sources.  Following  are  details  for  planning  data  colleo 
tion  from  each  of  these  critical  sources. 


2.1.1  Develop  Progress  Evaluation  Plan 

In  the  introduction  a number  of  specific 
requirements  and  functions  of  toe  internal 
evaluator  were  discussed.  These  .ocluded  the 
collection  and  interpretation  data  from 
students,  instructors,  project  schedules. 
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aru^  other  sources.  HoMever,  these  procedures  could  not  have  been  done 
properly  If  the  Internal  evaluator  had  not  been  Involved  the  instruc- 
tion fr<Mn  the  tine  It  was  begun.  Even  though  the  model  shows  Phase  V as 
the  last  phase,  this  Is  not  Intended  to  mean  that  this  function  will  not 
be  iiorforaed  or  planned  until  everything;  else  has  been  done.  Inter.tal 
evaluators  n:u^>t  be  a part  of  the  ISD  team  from  the  ':!9l nnl ng . 

Two  important  parts  of  the  evaluation  plan  must  be  prepared  bfl'cre 
aro'  substantial  work  has  been  done  on  the  course.  The  first  c. f tnese  is 
the  Progress  Evaluation  Plan.  This  Plan  will  state  how  the  evaluator 
will  prepare  reports  to  management  on  tlic  progress  of  the  ISO  effort, 
what  Is  the  mlnimian  amount  of  Information  that  management  must  have  to 
make  effective  decisions,  and  how  freq»«ntly  tills  InTormatlon  must  be 
provided.  The  plan  siiould  contain  well  summarized  Information  and  be 
reported  as  Infrequently  as  possible  In  order  to  avoid  unnecessary  paper- 
work. 

Step  one  Is  to  find  out  from  tfie  managei*s  what  they  believe  their 
information  requirements  to  be.  Usually,  the  best  approach  is  to  suggest 
alternative  kinds  of  information  ind  a reporting  schedule,  making  sure 
that  nothing  in^ortant  has  been  left  oat.  Some  managers  will  identify 
key  checkpoints  in  the  process  and  delegate  authority  to  nis  subordinates 
to  accomplish  these  efforts,  ite  may  want  to  be  notified  only  on  t^e 
exceptions  to  the  plan;  that  is,  when  things  arc  not  working  according 
to  schedule. 

Stej>  two  is  for  the  prtrject  pianner^  and  evaluators  to  prepare  a 
project  time  schedule  and  Idfintify  the  sources  of  the  information  required 
by  the  manager.  The  scheciulf  should  identify  events  that  are  to  be  completed 


13 


at  certain  sp/icific  tines.  The  evaluator  nust  find  out  who  can  make  the 
stat^ent  that  ai\y  or  all  parts  of  a block  are  con^leted.  If  the  evalua- 
tor then  kiMMs  what  the  steps  are  and  who  can  authenticate  their  completion, 
two  major  steps  have  been  accoa^)11shed.  He  will  be  able  to  ^t  data  and 
report  progress  In  a form  that  Is  most  useful  to  the  manager.  Normally, 

Bivy  deviation  from  the  plan  should  be  explained  to  the  manager  and  a dis- 
cussion about  how  to  eliminate  the  discrepancy  should  produce  an  acceptable 
solution.  Figure  V.2  Is  an  exan4;t£  of  part  of  a project  schedule. 


Estimated  Actual 


Event  No . 

Event  Name  Activity  Name 

Completion 

Completion 

Notf 

001 

Begin  Block  I.l  Conduct  Job 
Arialysis 

1 May 

(1) 

002 

Begin  Block  1.2  Select  Tasks/ 
Functions 

1 Sept 

22  Oct. 

(2) 

003 

Begin  Block  1.3  Analyze  Per- 
formance 
Requi rements 

1 Oct. 

22  Nov. 

(2) 

004 

Begin  Block  1.4  Analyze  Existing 
Course 

10  Jan 

15  Feb 

(3) 

oi9 

Begin  Block  V.3  Revise  Instruc- 
tional System 

15  Aug 

1 Sept 

(15) 

Notes: 


(1)  Pending  Approval  from  Comnand  HQ;  expected  15  April. 

(2)  Travel  funds  delayed;  field  trip  not  authorized. 

(3)  Consultant  hired;  unsure  of  arrival 

(15)  Printing  (teadlines  arranged  from  Conwand  HQ 


FIGURE  V.2:  Exaj^le  of  a Reporting  Form  to  Maintain 
Control  of  a Project  Schedule 
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This  report  c£in  becone  very  lengthy  and  con^Hcated  on  large  scale 
projects  and  Mill  require  people  or  a ccNiputer  to  keep  It  up  to  date. 

Several  references  on  project  scheduling  and  reporting  are  Include  at 
the  end  of  this  block.  Hapy  commands  have  (teve loped  their  own  highly 
refined  reporting  systens  for  similar  purposes, 
can  be  readily  adapted  to  ISO  procedures. 

2.1.2  Develop  Process  Evaluation  Plan 

The  second  major  effort  of  the  Internal 
evaluator  In  the  early  stages  of  the  ISO 
process  Is  the  preparation  of  the  Process 
Evaluation  Plan.  Mainly,  this  consists  of 
going  through  the  various  steps  of  the  model 
and  Identifying  which  of  the  steps  and  pro- 
cedures In  the  process  will  be  used  for  the 
course  under  development.  The  purpose  of  the  process  evaluation  Is  to 
describe  ■’.nd  docianent  the  actual  developmental  process  for  this  particular 
Instruction.  These  data  are  found  in  the  output  documentation  at  the  end  of 
each  block.  If  any  changes  were  made  in  the  standard  procedure,  they  should 
be  docunented.  It  is  useful  to  Indicate  why  any  processes  and  steps  were 
not  completed  as  required,  so  that  in  the  future  those  with  responsiuility 
for  the  course  will  have  a better  understanding  of  what  was  done. 

The  procedure  consists  of  preparing  a checklist  for  each  block  In 
the  model.  The  primary  internal  steps  necessary  to  produce  the  outp.it  of 
the  block  should  be  listed.  Notations  of  exceptions  should  be  rcque5.ted 
iii  each  block.  F^gu^e  V.3  is  an  example  of  a form  that  might  be  used.  The 
sample  form  also  contains  some  example  entries. 


These  existing  plans 
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Block  I.l:  Analyze  Job 

Process  Evaluation  Checklist 

Completea  Procedure?  Explanation 

Review  Available  Job  Data 

Yes 

Plan  Data  Collection 

Yes 

Train  Job  Analysts 

No 

Experienced  JA’s  avail. 

Prepare  Forms 

No 

Used  existing  form. 

Select  Sample 

Yes 

Collect  Bata  at  Job  Site 

No 

Travel  funefe  unavail- 
able; questionnaire 
only  was  used  for 
task  list. 

Analyze  Data 

Yes 

Revise  Consolidated  List 

Yes 

Based  on  questionnaire 
responses  only. 

FIGURE  V.3:  Sanple  Process  Evaluation  Checklist  for  Block  I.l. 


Any  suitable  fom  can  be  used,  or  the  data  can  be  sumoiarized  from  the 
output  of  each  block.  The  principal  information  is  the  explanation  of  any 
alternative  procedure  followed,  so  that  future  managers  will  know  what 
has  and  has  not  been  done. 

If  such  a process  evaluation  instru>^nt  had  been  used  In  the  exanple 
of  the  regulation  mentioned  in  the  Introduction  to  this  block,  the  nega- 
tive aspects  of  the  experience  might  have  been  virtually  eliminated. 

I While  the  preparation  of  both  process  and  progress  evaluation  plans 

I and  the  monitoring  of  these  efforts  are  functions  that  should  be  performed 

I because  of  the  organization  of  many  commands,  it  Is  not  always  reasonable 

I 
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to  assign  these  functions  to  the  Internal  evaluator.  While  the 
data  are  basically  evaluative  In  nature,  they  may  be  mor^  properly 
collected  within  the  management  system  rather  than  In  the  evaluation 
system. 

The  process  evaluation  procedures  should  be  extended  to  Includf.^ 
the  outputs  from  each  of  the  blocks  of  the  Model.  Each  block  has  a 
specified  output  which  Is  required  by  succeeding  steps  of  the  Model 
In  order  to  keep  the  development  going.  Each  of  these  outputs  must  be 
examined  and  a judgement  made  as  to  their  adequacy;  they  are  either  in 
conformance  with  tiie  needs  and  requirements  of  the  system,  or  are 
<tef1c1ent  becaime  they  lack  certain  specific  identifiable  features. 

While  It  Is  not  possible  for  an  evaluator  to  tell  for  sure  whether  the 
procedures  for  a proper  job  analysis  wer^  followed  during  the  collection 
of  the  data,  he  can  determine  whether  the  output  of  Block  I.l  is 
sufficient  to  do  the  work  called  for  In  Block  1.2.  If  the  output 
of  Block  1.2  is  not  con^lete,  more  JPIte  may  be  developed  than 
necessary  in  Block  1.3,  etc.  The  process  evaluation  should  indicate 
whether  the  outputs  of  all  the  blocks  are  of  sufficient  clarity  and 
quality  to  permit  the  continuation  of  the  process.  Outputs  found  to 
be  lacking  need  to  be  identified  and  revised  prior  to  continuation. 

The  ISO  process  generates  a large  nimiber  of  products,  just  as 
any  instructional  method  does.  Often,  it  may  seem  that  the  ISO  process 
is  more  conplicated  and  requires  more  effort  than  the  existing  system. 
What  tends  to  be  disregarded  is  that  all  of  the  rules  and  proc-idures 
for  operating  the  present  system  have  not  been  carefully  written  down 
and  docunented.  Much  of  the  expertise  lies  in  the  knowledge  and 
understanding  of  the  people  who  are  in  charge  of  the  courses.  If  all 


of  that  Infomation  and  procedure  were  gathered  Into  a Si-i  of  Instruc 
tioiial  manuals,  it  probably  would  seem  at  least  as  complicated  as  the 
ISO  process  does  to  those  gcing  through  it  for  the  first  time. 

Table  V.2  provides  a list  of  dociaaentation  which  usually  results 
from  application  of  the  model  and  can  be  used  to  develop  a con^lete 
course  history.  On  the  basis  of  this  existing  data,  a reasonably 
accurate  baseline  can  be  established. 


TABLE  V.2 

Required  ISO  Documentation  by  Block  for  the  Process  Evaluation 


Block  I.l: 

1.  Request  for  training  or  change  in  training 

a.  change  in  requirements  for  existing  job 

b.  new  job 

c.  new  equipment 

d.  quality  control 

2.  Definition  of  job 

3.  Description  of  job  task  analysis 

4.  Questionnaires  with  memory  statement  of  reponses 

5.  Validated  task  list 

Conditions,  cues,  standards,  and  elements  for  each  task 
selected  for  training 

Block  1.2: 

1.  Criteria  for  evaluating  tasks 

2.  Survey  sources 

3.  Collection  forms  with  simrary  of  data 

4.  Data  analysis 

5.  Hanagement  inputs 

6.  List  of  tasks  selected  for  training 

Block  1.3: 


1.  JPHs  with  administrative  instructions 

2.  Field  test  data  or  JPHs 


'JlixillilUlllllHlI 


18 

Block  1.4: 

1.  Inscription  of  existing  courses  with  ar\v  or  all  of  the 
following  that  are  available 

a.  Job  analysis 

b.  tasks  Included  In  training 

c.  performance  measures 

Block  1.5: 

1.  Job  performance  measures  grouped  by  possible  settings 
and  costs  for  each  setting 

2.  Selected  setting(s) 

Block  II. 1: 

1.  Learning  task  analysis 

2.  Learning  objectives 

Block  II. 2; 

1.  Entry  tests 

2.  Pretests 

3.  Posttests  within  course  tests 

4.  Posttests 

5.  Testing  plan 

Block  11.3: 

1.  Assunptlons  about  entry  behavior 

2.  Data  from  validation 

3.  Description  of  target  population 

Block  II.4: 

1.  Description  of  sequencing  problems 

2.  Alternative  sequencing  problems 

3.  Sequenced  list  of  objectives 

Block  Ill.l: 

1.  Objectives  classified  by  learning  categories 

2.  List  of  learning  guidelines  and  leaning  activities 

Block  111.2: 

1.  Criteria  for  selecting  delivery  system 

2.  Selected  delivery  system 

3.  Selected  management  plan  and  rationale  for  choice 

4.  System  Master  Plan 


i 
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Block  II1.3: 

1.  Ascription  of  review  process 

2.  Selected  materials  with  available  data 

Block  HI. 4: 

1.  Ata  from  development  trials 

2.  Technical  information  such  as: 

a.  film  type 

b.  light  levels 

c.  narrator,  etr. 

3.  Art,  instruction,  scripts,  artwork,  story  boards,  etc. 

4.  Revisions  ^rom  III. 5 

5.  Final  product 

Block  IV. 1: 

1.  Ascription  of  supplementary  instruction  given  to 
instructors  and  students 

2.  Special  training  of  instructors  and  instructor  qualifications 

3.  Description  of  student  population 

4.  Aquired  remedial  work 

5.  Outline  of  time  allocation  plan,  facility  and  equipment 
requi rements 

Block  IV. 2: 

1.  Instructors'  log  books 

2.  Instructors'  lists  of  stuAnt  problems 

3.  Times  required 

4.  Any  Aviations  from  course  plan 


The  processes  specified  for  a particular  course  are  difficult  to 
evaluate  directly  because  they  may  occur  over  a long  period  of  time 
and  observing  them  may  be  expensive  and  difficult.  However,  it  should  be 
possible  to  Avelop  a checklist  which  is  procedurally  sound  that  is 
unique  to  each  of  the  blocks  and  which  suinnarizes  what  has  occurred.  It 
could  also  be  in  the  form  of  a questionnaire  or  interview.  The  purpose  of 
the  process  evaluation  is  to  document  the  authenticity  of  the  application 
of  the  model.  If  the  course  does  not  work,  it  is  much  easier  to  revise 
if  one  can  assume  in  the  beginning  that  it  was  developed  according 


to  ft  standard  empirical  procedure.  See  Figure  V.4  for  a sample  checklist. 


Checklist  for  Product  Evaluation 
Block  II. 4 

Describe  Entry  Behavior 


Coinponent 


1.  Are  administrative  requirements  specified? 

a.  Length  of  service  obligated 

b.  Rank/Rating 

c.  Prerequisites  described 

d.  Unit/ fleet  experience 

2.  Are  physical  requirements  specified? 

a.  Vision 

b.  Hearing 

c.  Size 

d.  Coordination 


3.  Are  academic  requirements  specified? 

a.  Basic  Training 

b.  Initial  E-s  --  E-4  Training 


4.  Are  aptitude  scores  specified? 

a.  AFQT,  other 

b.  Language,  math,  science 

c. 

5.  Entry  Behavior 

a.  Arithmetic 

b.  Welding 

c.  Soldering 

d.  Hand  tools 

e.  Power  tools 

f.  Other  relevant  requirements 


FIGURE  V.4:  Sample  Product  Evaluation  Checklist  for 

Product  of  II. 3:  Describe  Entry  Behavior. 

The  information  can  ordinarily  be  suirmari zed 
from  documentation  produced  in  each  block. 
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If  revisions  are  begun  In  the  Instructional  materials  but  the 
principal  probleos  lie  In  the  Inattequacy  of  the  JPH$»  many  resources 


will  be  wasted.  The  evaluation  of  the  process  of  ISO  course  developitent 
Is  Intended  to  avoid  useless  "paper  exercises".  If  the  procedures 
cannot  be  followed  by  people  adequately  supervised  and  trained,  there 
are  severe  problems  with  the  requirements.  Perhaps  the  procedures  should 
be  modified,  not  the  people  who  do  the  work.  The  Implications  of  this 
kind  of  evaluation  are  profound.  It  means  that  each  person  Involved 
mist  have  adequate  training  so  he  will  know  how  to  do  what  Is  expected. 
Further,  he  imist  have  adequate  supervision  to  resolve  any  problens 
encountered. 

If  for  any  reason  a specified  procedure  cannot  be  followed,  that 
fact  should  be  docimented  and  explained.  For  exanple.  In  Block  I.l, 
ANAIY2E  JOB,  a number  of  actions  are  specified.  These  Include  inter- 
views with  supervis;:'rs  and  job  Incumbents  and  the  development  and 
circulation  of  questionnaires.  If  there  are  no  job  Incunbents  locally 
and  there  are  lo  travel  funds,  this  step  cannot  be  followed  accurately. 

The  circumstances  should  be  adequately  docimiented  so  that  later  decisions 
can  take  Into  account  the  constraints  in  effect  at  the  time  the 
instructional  development  was  started. 

If  the  constraint  is  properly  documented,  management  can  make  more 
reasonable  decisions  about  the  proper  course  of  action  if,  upon  canpletlon 
of  the  course,  the  graduates  canrwt  perform  on  the  job.  They  will  know 
not  to  revise  the  existing  course  before  correcting  the  original  In- 
adequate job  analysis. 
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In  Block  1.2,  SELECT  TASKS  FOR  TRAINING,  9 number  of  safeguards  have 
been  built  In  to  make  sure  that  every  person  in  the  DOS  Vi'o  performs  a 
given  job.  no  matter  on  what  ship  or  station,  sas  been  represented  in 
the  sampling  and  rating.  If  a significant  segment  of  the  DOS  has  not 
been  san^led,  then  graduates  reporting  to  those  ships  or  stations  may 
be  inadequately  trained.  It  was  not  the  quality  of  the  training,  per  se, 
but  the  tasks  selected  for  training  that  were  not  adequate.  In  the  real 
world,  it  is  often  necessary  to  proceed  with  inadequate  or  Incomplete 
information.  One  can  wait  only  so  long  and  urge  only  so  often,  for 
example,  that  the  questionnaires  be  completed  and  retiirned.  But  the 
discrepancy  between  the  requirement  and  the  results  should  be  documented 
in  order  to  avoid  making  a bad  inference  about  the  cause  of  the  problem. 

It  3 generally  recognized  that  people  a' e far  more  willing  to  admit 
their  errors  and  shortcomings  to  impartial  an  disinterested  third  parties. 
Hopetully.  the  evaluators  can  collect  the  data  without  having  to  find  some- 
one to  blame.  There  is  a great  difference  betw?en  making  a clear  state- 
ment of  a problem  and  trying  to  find  someone  whc  is  at  feult. 

2,1,3  Develop  Performance  Evaluation  Plan 

The  principal  source  of  information  about 
an  instructional  program's  effectiveness,  that 
is,  its  ability  to  accomplish  the  objectives, 
is  from  the  students  who  receive  the  instruc- 
tion. A clear  and  specific  distinction  is 
made  here  between  the  doctri'  and  the  results. 


Doctrine  can  only  be  determined  by  experts  in 


23 

the  field.  Students  are  not  very  useful  In  this  area.  Whatever  the 
doctrine,  good  or  bad,  accurate  or  Inaccurate,  the  students  can  provide 
information  on  what  they  learned.  Student  performance  data  can  be 
collected  only  from  students.  SMEs  cannot  supply  it.  No  revisions 
should  be  undertaken  to  correct  methodology  without  adequate  student 
performance  data.  Revisions  take  time  and  money  and  should  not  be  done 
unless  there  is  good  evidence  that  the  revisions  are  necessary.  This 
rule  does  not  apply  to  revisions  undertaken  to  correct  errors  in  doctrine. 

There  are  four  areas  that  are  irportart  to  consider  in  evaluating 
students: 

1.  External  requirements 

2.  Entry  skills 

3.  Performance  on  internal  tests 

4.  lime  required  to  complete  instructional  units 

For  all  OT  these  areas,  appropriate  forms  should  be  us.  d for  collecting 
the  data. 


2. 1.3.1  Exiern-il  Requirements 
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course  designers.  It  is  fairly  easy  to  design  a form  that  will  contain 
all  the  useful  Information  from  the  student's  prior  history;  all  of  the 
Items  nentloned  In  Block  II. 3.  This  form  Is  not  significnatly  different  | 

from  a Job  application  blank. 

Students  probably  should  never  be  rejected  from  a course  for  falling 
to  meet  assumed  or  specified  prerequisites  until  real  data  from  course  ! 

operations  confirm  or  reject  the  prerequisite  criteria.  There  is  no 
way  of  knowing  that  the  prerequisites  are  actually  necessary  until  it  I 

can  be  shown  that  students  having  the  prerequisites  tend  to  succeed  while 
students  not  having  them  tend  to  fall  to  reach  mastery  in  a reasonable 
time  during  instruction. 

2. 1.3.2  Entry  Skills 

If  students  have,  or  do  not  have,  the  external  prerequisites,  they 

may  still  have  or  not  have  the  entry  skills.  The  entry  skills  determi-  J 

I 

nation  is  Inportant  to  know  whether  to  place  the  students  at  the  beginn-  j 

I 

ing  or  provide  preliminary  instruction.  Entry  skills  are  very  specific,  | 

measur*eable  behaviors  that  have  been  determined,  through  the  process  of  j 

analysis  of  learning  requirements,  to  be  basic  to  acquisition  of  sub-  1 

sequent  knowledge  or  skill  in  the  course.  Pretests  are  used  to  decide 

whether  tr  start  the  student  in  an  advanced  unit.  Since  some  students 

will  know  much  more  about  the  subject  matter  than  others,  they  may  be 

advanced.  This  process  Is  described  in  Blocks  II. 2 and  II. 3. 

The  determination  of  entry  skills  is  made  on  the  basis  of  the  student's 
ability  to  answer  questions  or  demonstrate  adequate  performance  on  the 
knowledge  and  skill  needed  prior  to  instruction.  It  is  not  an  aptitude 
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or  Intelligence  test,  but  an  instruction-related  test  based  on  applicable 
learning  objectives. 

In  some  schools,  it  is  possible  to  ask  students  whether  or  not  they 
know  anything  about  Liie  in.. tniction.  If  they  say  they  do  not  and  their 
service  record  confirms  the  claim,  it  is  probably  reasonable  to  assume 
they  are  right.  If  they  say  they  do  know  something  about  the  subject, 
it  is  probably  worthwhile  to  administer  the  pretests  to  determine  how 
much.  Usually  anything  the  student  already  knows  should  not  be  taught 
him  again. 

The';’  is  a difference  between  a pretest  and  a test  of  entry  skills. 

The  pretest  Is  to  determine  what  the  student  knows  about  the  specific 
unit  to  be  studied.  The  entry  skills  test  Is  to  determine  if  the  student 
has  mastered  skill  or  knowledge  that  is  basic  to  the  new  instruction  but 
is  not  intended  to  be  a part  of  the  instruction.  Jn  this  situation,  use 
the  test  developed  for  this  purpose  as  described  in  Block  11.3.  Normally, 
these  tests  will  be  administered  by  the  instructors  who  have  been  desig- 
nated for  that  assignment.  Keep  in  mind  that  the  purposes  of  the  internal 
evaluator  and  the  purposes  of  the  instructionol  staff  are  not  exactly  the 
same. 

2. 1.3. 3 Performance  on  Within-Course  Tests  and  Posttests 

The  internal  evaluator  will  not  have  to  be  so  much  concerned  about 
development  of  special  tests  or  evaluation  instruments  as  he  will  the 
systematic  collection  and  accumulation  of  the  data  collected  routinely 
during  the  instruction.  In  Block  III. 5,  the  validation  process  yielded 
useful  information  about  the  instructional  materials  based  on  the  tests 
administered  during  student  trials.  At  some  point  the  early  trials  or 
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iterations  of  a course  become  the  baseline  data.  Enough  data  is  ccTiected 
to  be  able  to  say  what  the  results  were  when  the  course  started.  Every- 
thii^  else  can  be  measured  from  that  point. 

There  are  many  sources  of  performance  data  in  the  ISO  model.  Host 
of  this  data  is  collected  in  Block  IV.2:  CONDUCT  INSTRUCTION.  The  Internal 
evaluator  uses  the  same  performance  data  to  evaluate  the  instruction  that 
the  instructors  or  monitors  use,  to  evaluate  the  students.  It  is  simply 
used  in  a different  manner.  Block  III. 5 presents  an  approach  to  the 
collection  and  display  of  performance  data  for  a number  of  students  on 
a nifiii>er  of  objectives. 

Technically,  in  this  section  of  this  block  the  concern  is  with  the 
eval'dation  procedures  for  students,  and  not  for  the  instruction  as  a 
whole.  The  instructional  staff  or  the  test  administrator  will  have 
administered  the  proper  test  to  the  students  and  decided  which  of  the 
students  met  the  requirements  and  which  required  additional  instruction. 
Ricording  this  information  for  each  of  the  major  subdivisions  of  the 
instruction  can  be  most  helpful.  ProbeMy  the  greatest  benefit  is  to 
allow  students  to  continue  in  the  program  regardless  of  their  test  score 
on  the  go/no-go  tests  until  it  has  been  clearly  established  whether  those 
who  are  “go's"  can  continue  to  do  the  work  and  those  who  are  "no-go's" 
cannot. 

Following  instruction  of  any  kind,  there  should  be  a posttest.  Post- 
tests may  be  given  on  practice  exercises,  lessons,  modules,  units,  or 
complete  courses.  They  should  be  as  authentic  as  possible;  that  is, 
they  should  be  the  actual  task  or  close  approximations. 
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2. 1.3,4  Time  Required  to  Complete  Instructional  Units 

One  of  the  more  valuable  types  of  data  that  can  be  collected  by  the 
Instructional  stoff  Is  the  amount  of  time  required  to  complete  each  of 
the  d)jectives.  or  instructional  units.  While  there  may  be  some  rela- 
tionship betw^n  successful  performance  and  the  amount  of  time,  the 
major  value  of  tiie  time  data  conws  when  alternative  means  of  reaching 
the  objectives  need  to  be  compared  with  the  original  plan.  Some  instruc- 
tional methods  are  virtually  equal  in  effectiveness,  but  they  are  unequal 
in  time  required  to  con^ilete  them.  Consider  the  block  scheduled  lecture 
approach  where  each  student  spends  the  same  amount  of  time  in  the  presence 
of  instruction.  There,  students  spend  approximately  equal  time  in  instruc 
tion,  not  including  the  outside  time  they  spend  studying.  If  another 
method  was  considered.  It  would  be  useful  to  have  the  time  data  for  the 
other  method  to  make  the  comparisons. 

Student  time  cards  can  be  prepared  if  there  are  facilities  for  using 
them,  and  the  time  data  summarized  off  the  time  cards.  Sometimes  it  is 
possible  to  use  sign-in  sheets  and  student-kept  time  logs.  The  latter 
are  not  so  accurate  as  the  former,  but  with  enough  students  they  ai'e 
likely  to  serve  the  purpose  quite  well,  particularly  on  long  courses. 

Time  records,  except  those  requiring  expensive  equipment  and  those  kept 
in  a very  tightly  controlled  setting  are  not  likely  to  be  precise.  But, 
like  so  many  other  kinds  of  measures,  they  may  be  the  best  available  and 
will  have  to  be  used. 
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Time  should  be  kept  on  the  smallest  practical  unit  of  instruction. 
Also,  wherever  possible,  time  should  be  limited  to  actual  instruction, 


Indepenitent  of  waiting,  transportation,  actninistra :1ve  sntters.  and  other 
Interferences,  except  where  the  unused  time  Is  an  Integral  oart  of  the 
course  as  deslgr^d.  If  stu<tents  must  spend  so  much  time  doing  actnlnls- 
tratlve  duties  and  would  have  the  ^ame  obligation  under  any  other  form 
of  Instruction,  there  is  little  point  in  counting  those  duties  as  Instruc- 
tional time.  If  the  time  Is  to  be  recorded  and  kept  by  the  student,  It 
Is  imperative  that  a system  b«  established  to  collect  the  student  records 
at  frequent  intervals. 

Perhaps  students  must  turn  In  their  time  sheets  prior  to  getting  new 
materials,  passing  to  the  next  unit,  leaving  on  Friday,  or  getting  a new 
chow  card.  This  turning  in  of  time  cards  and  time  sheets  will  not  happen 
without  careful  management.  Often  people  mistake  the  intention  of  the 
time  records  and  believe  that  it  Is  a way  "the  Man"  has  of  checking  up 
on  what  they  are  doing.  Time,  collected  as  a pact  of  an  evaluation  effort, 
should  not  be  used  for  administrative  purposes.  If  it  is,  the  records 
will  be  even  less  accurate  than  they  would  be  if  students  keep  them  with- 
out this  threat. 

Time  is  probably  best  recorded  on  a "per  unit  of  achievement"  basis 
rather  than  on  a calendar  basis.  It  is  not  of  great  value  to  know  that 
on  Thursday,  Jones  s|«nt  forty  minutes  watching  a film.  It  is  more 
Important  to  find  out  how  much  tinvi  Jones  spent  watching  film  to  accom- 
plish each  specific  objective. 

The  kind  and  quality  of  the  time  records  kept  wll^  depend  on  the 
emphasis  placed  on  efficiency  by  management,  the  availability  and  assign- 
ment of  people  to  do  the  job,  and  the  ability  of  the  course  manager  to 
secure  the  cooperation  of  the  students,  particularly  If  they  will  be  keep- 
ing their  own  tinw  records. 
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2.1.4  Develop  Plan  for  Collecting  Information  Fran  Studenus 


Students  can  be  the  source  of  useful  Q 

Information  about  the  perceived  quality  j 

and  preferability  of  Instructional  events  ,1  ^ 


and  materials.  Usually,  this  information  j — 1 _ 

is  collected  on  questionnaires  and  rating  j ~i 

scales.  Typically,  the  questionnaires  and 

rating  scales  are  prepared  1n  advance  for  i 

each  of  the  events  and  materials  to  be 
rated.  Ratings  can  be  on  an  absolute  basis: 

1.  What  did  you  think  of  the  film? 

2.  Which  did  you  like  better,  film  A or  film  B? 

3.  Would  you  prefer  to  practice  longer  before  you  take  the 
test,  or  did  you  have  enough  time? 

And  other  questions  of  that  nature.  Some  examples  of  questionnaires  and 
rating  sheets  are  Included  in  Appendix  A. 

Students  can  report  on  problems  they  encounter  with  the  instruction, 
the  Instructors,  evaluators,  or  other  materials  and  people  associated 
with  the  instruction.  These  reports  are  best  prepared  on  a regular  basis 
and  as  close  In  time  to  the  event. reported  upon  as  possible  in  order  to 
avoid  forgetting  of  critical  details.  The  method  of  critical  incidents 
is  particularly  suitable  for  obtaining  specific  positive  and  negative 
incidents  in  the  instructional  program.  Sumnarizing  these  incidents  can 
provide  good  int'ormation  about  the  course.  If  the  incidents  are  to  be 
written,  some  consideration  should  be  given  to  the  ability  of  the  trainee 
to  compose  narrative  prose. 


Students  can  be  used  to  rate  Instruction,  their  Instructors,  and  their 
classmates.  There  can  be  no  relationship  between  the  student's  rating  of 
the  instructor  and  his  gratte  in  the  class.  Ratings  roust  be  carefully 
collected.  The  student  should  not  be  identified  as  the  rater.  Further, 
the  ratings  should  be  seen  and  presented  in  a constructive  manrar;  that 
is,  they  should  not  be  seen  by  the  stuctent  as  a way  to  "get  back"  at  an 
unpopular  instructor.,  The  purpose  of  ratings  of  instructors  is  to  find 
specific  points  in  the  instructor's  behavior  which  can  or  should  be  changed 
or  emphasized.  If  he  is  doing  something  right,  it  is  good  that  he  inows 
that,  and  the  same  applies  if  he  is  ckiing  something  wrong. 

Probably  the  greatest  difficulty  with  rating  forms  is  the  tender  :y  to 
include  questions  on  the  rating  form  that  those  who  use  it  are  not  quali- 
fied to  answer.  Just  as  people  untrained  in  medicine  should  not  be  asked 
to  rate  the  health  of  an  individual,  students  should  not  be  asked  to  rate 
the  instructor's  knowledge  of  a subject  matter.  The  ratings  should  bear 
a close  relationship  to  what  the  student  has  had  the  chance  to  observ:;, 
and  the  ratings  should  be  based  on  that  observation.  It  is  not  a good 
idea  to  have  students  try  to  compare  Instructors  on  a relative  basis 

Students  can  tell  whether  they  like,  dislike,  or  are  Indifferent  to 
instruction  they  have  recently  experienced.  They  may  not  be  able  to  say 
why!  What  students  should  be  asked  are  specific  personal  experiences 
and  their  feelings  or  opinions  about  these  experiences.  Granting  student 
preferences,  all  other  things  being  equal,  probably  will  increase  morale. 
Further,  the  filling  out  of  attitude  scales  and  questionnaires  has  been 
known  to  reduce  anxiety  in  courses  that  historically  have  created  much 
concern ''and  anxiety. 
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Instructors  are  in  a unique  position  to 
provide  much  valuable  information  about  the 
instruction.  One  of  the  more  important  kinds 
of  information  that  can  be  obtained  from 
instructors  is  doc'.’ine.  Rarely  will  instruc- 
tors be  assigned  to  a course  if  they  have  no 
knowledge  of  the  doctrine  and  content  being 
taught.  As  many  people  who  have  taught  know, 
many  problems  are  identified  in  the  instructional  materials  and  procedures 
when  they  are  used  with  students.  Students  have  a way  of  asking  "How" 
or  "Why"  to  ideas  and  procedures  taken  for  granted  by  those  in  the  field. 
Answering  these  hard  questions  has  helped  a number  of  instructors  learn 
much  more  about  their  own  discipline  by  having  to  explain  clearly  to 
students. 

Another  source  of  data  which  can  come  only  from  instructors  or  test 
administrators  is  the  performance  of  students  on  the  objectives  being 
taught.  It  is  normally  the  instructor  or  test  administrator  who  gives 
the  performance  tests  or  other  measures  to  determine  when  the  students 
have  mastered  the  objectives.  Further,  the  instructor  can  supplement 
this  hard  data  with  ratings  of  students  on  other  important  variables, 
including  nwtivation,  application,  effort,  and  ability  to  follow  instruc- 
tions . 

If  the  instructors  are  asked  to  keep  logs  or  notes  on  specific  prob- 
lems which  occur  in  the  course,  these  can  beconv?  valuable  when  analyzed. 
The  very  act  of  recording  often  focuses  attention  on  the  problem  and 
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solutions  from  conversations  among  those  familiar  with  the  problems. 

If  Instructors  are  asked  from  time  to  time  to  prepare  critical  incidents 
highlighting  particularly  successful  and  unsuccessful  instructional  events 
or  procedures,  the  analysis  of  these  incidents  can  yield  valuable  infor- 
mation. The  analysis  cam  perhaps  be  better  used  to  improve  instruction 
than  can  any  other  kind  of  information. 

In  many  cases,  it  will  be  possible  for  the  instructor  to  keep  reasonably 
accurate  time  data.  He  will  be  able  to  record  the  beginning  and  ending 
time  for  all  stuctents  on  each  objective.  While  this  data  collection  takes 
time.  It  can  be  some  of  the  more  valuable  information  accimiulated.  Ordi- 
narily, Instructors  will  be  able  to  keep  this  kind  of  time  data  only  in 
those  situations  where  the  students  are  self-paced,  or  individually 
scheduled  on  the  objectives  in  question.  It  Is  extremely  difficult  to 
collect  accurate  time  data  on  groups,  unless  one  is  simply  recording  hours 
of  "exposure." 

The  more  detail  that  can  be  included  with  time  data,  the  better.  If 
the  instruction  involves  use  of  manuals,  tools,  training  devices,  labora- 
tory equipment,  presentations,  and  practice  exercises,  these  times  should 
be  kept  separately.  Later,  times  can  be  compared  to  the  performance  data 
and  the  student  impressions  of  the  value  of  the  different  experiences. 

Often  it  is  desirable  for  Instructors  to  observe  and  record  evaluative 
ratings  of  peers  as  they  go  about  their  jobs.  This  information,  if  pro- 
vided in  the  form  of  feedback,  can  be  helpful  in  improving  instructor 
performance . 
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EXAMPLE 

In  the  a(te1n1strat1on  of  performance  tests.  If  the 
Instructor  is  asking  the  san«  questions  and  requiring 
the  same  performance  of  all  of  the  trainees,  this 
Information  may  quickly  spread  among  the  trainees  who 
may  try  to  "beat”  the  system.  The  instructor  may  r»t 
have  intentionally  followed  the  same  procedure  or  may 
be  unaware  that  he  is  giving  the  answers  away  by  habitual 
practi  ces . 

Observation  of  the  instructor  by  other  informed  instructors  can  be 
helpful.  Care  should  be  taken  not  to  ask  questions  on  the  questionnaire 
which  cannot  be  rated  by  observers,  e.g.,  "Knowledge  of  subject." 

Internal  evaluators  may  find  some  of  these  data  much  more  useful  than 
other  data.  The  value  of  the  Information  will  be  determined  in  part  by 
the  kinds  of  problems  that  the  course  is  experiencing.  The  problems  will 
be  different  for  old  instruction  that  has  had  most  of  the  problems  removed 
from  it  than  for  new  instruction  that  has  only  recently  come  under  the 
scrutiny  of  the  evaluators.  If  the  course  has  already  been  determined 
effective,  the  principal  concern  may  be  the  collection  of  data  that  will 
increase  efficiency.  It  may  be  that  more  students  could  be  sent  through 
the  program  without  increasing  the  number  of  people  required  to  manage 
it,  or  it  may  be  that  certain  people  involved  with  the  course  can  be 
reassigned  when  t^_  course  becomes  more  efficient. 

2.2  Conduct  Internal  Evaluation  ~| 


If  the  internal  evaluation  has  been  ! 

appropriately  planned;  if  all  forms,  tests. 



instructions  fcr  use,  and  trained  personnel 

f 

are  avail  able,  the  actual  conducting  of  the 
evaluation  is  largely  a matter  of  carefully 
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following  the  plan.  However,  critical  steps  in  a program  do  not  just 
happen."  Prepare  for  the  unexpected.  Be  prepared  to  modify  or  add  to 
the  plan  as  instruction  proceeds.  Apy  deviations  from  plan  must  be 
carefully  documented  along  with  the  reasons  why  the  changes  were  made. 
Collecting  too  much  information  costs  time  and  money.  But  not  collect- 
ing essential  Information  will  cost  even  more. 

2.3  Interpret  Data  and  Hake  Recommendations 
The  primary  product  of  this  block  is 
an  internal  evaluation  report  (1N£R)  This 
report  Is  a summary  statement,  referencing 
all  pert1ner>t  infonnation  resulting  from 
the  procedures  followed  in  this  block,  of 
the  internal  evaluation  findings,  their 
interpretation,  and  specific  recommenda- 
tions for  revision  of  the  instruction.  A summary  statement  of  the  previous 
sections  of  this  block  should  be  included  in  this  report..  Details  of 
appropriate  display  and  interpretation  of  data  follows. 

2.3.1  Display  and  Interpretation  of  General  Data 

After  the  evaluation  data  has  been  obtained  and  tabulated  in  a usable 
form,  it  is  time  to  begin  making  interpre '■.ations  and  drawing  conclusions. 

A major  function  of  the  internal  evaluation  process  is  to  suggest  what 
the  data  collected  might  mean;  that  is,  what  implications  it  might  have 
for  'Jlock  V.3:  REVISE  SYSTEM.  Those  who  are  well  trained  in  methods  of 
educational  and  training  research  will  be  better  able  to  manipulate  the 

data  in  a number  of  ways  that  will  provide  specific  cause  and  effect 
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relationships,  and  tentative  conclusions.  If  such  people  are  available 
in  your  cownand,  it  could  be  very  useful  to  involve  them  in  the  develop- 
ment of  data  analysis  and  Interpretation  studies. 

If  researchers  are  not  available,  the  next  best  method  1s  to  tabulate 

the  data  by  hand  in  useful  ways.  First,  the  data  should  be  categorized 

by  the  personal  characteristics  of  the  students;  the  measures  and  tests 

taken  before  they  arrived  at  the  school,  ihese  include  the  following: 

Age,  length  of  service 
Prior  training  in  this  area 
Experience  in  this  area 

Hobbies  and  personal  interests  (if  applicable) 

Scores  on  AFQT  and  other  Placen«nt  tests 

These  can  be  categorized  and  put  onto  a data  sheet  large  enough  to 
contain  all  of  the  background  information  and  all  of  the  information 
to  be  recorded  about  the  student  at  the  school.  Figure  V.5  provides  a 
suggested  form  of  a data  sheet  which  would  be  useful  for  hand  tabulation. 
If  you  have  a machine  capability,  you  should  consult  with  the  machine 
operators  to  be  sure  that  the  data  sheets  you  prepare  are  compatible  with 
the  machine  format.  Figure  V.6  displays  a summary  sheet  for  coded  data. 

In  Section  2.2,  the  performance  of  the  student  at  the  school  was 
carefully  recorded.  If  possible,  the  student's  actual  scores  on  the 
performance  measures  should  be  reported  here,  net  just  whether  he  was 
a "go  nr  no-go".  While  the  "go"  is  enough  for  the  student,  the  evaluator 
can  make  good  use  of  the  actual  scores  to  determine  the  instructional 
efficiency  of  the  modules  and  units.  It  is  true  that  some  performance 
measures  do  not  yield  a nuirAer  score,  and  those  will  have  to  be  recorded 
as  go/ no-go. 


i 

1 


2.  Length  of  Service  (months 


3.  Yea 


in  This  Area  (+  or  - 


5.  Hobbies  or  Personal  Interest  in  this 
Area  (•»■  or  - 


6.  Work  in  This  Area?  (+  or  - 


7.  School  Performance 


a.  Entry  Test  Score  (0-10 


b.  Modules/Units  Scores 


Student  No. 

9 ■>. 


21  18  19 


20 


1 

2 

3 

N 

c.  Modules/Units  Times  (total  hours) 

80 

90 

100 

85 

1 

10 

12 

14 

9 

2 

10 

12 

14 

9 

3 

9 

7 

9 

8 

N 

10 

8 

9 

9 

8.  Date  Departed 

5/74 

6/74 

6/74 

6/74 

9.  Assignment  to  DOS  (>  or  -) 

10.  Follow-Up  Information 

a.  Supervisor  Ratings  (1-5) 

b.  Performance  Test  Scones  (0-25) 

c.  Peer  Ratings  (1-5) 

d.  Promotion  Record  (numbered  promotions i 

e.  Safety  Record  (1-5) 


1 


FIGURE  V.5:  Sample  Coded  Data  Spread  Sheet 
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Sumnary  Sheet  of  Data 


I 


i 


1.  Age 

range 

years 

2.  Length  of  service 

range 

mos 

3.  Years  of  military  schooling 

range 

years 

4.  Training  In  this  area 

yes  % 

no % 

5.  Hobbles  or  personal  Interest  yes  % 

no % 

6.  Work  In  this  area 

yes  % 

no % 

7.  School  Performance 

a.  Entry  test  score 

range 

b.  Hodules/Units  scores 

(total) 

range 

1 

range 

2 

range 

3 

range 

N 

range 

c.  Hodules/Units 

Tiroes  (total) 

range 

1 

range 

2 

range 

3 

range 

N 

range 

8.  Date  Departed 

9.  Assignment  to  DOS 

yes  % 

no % 

10.  Follow-up 


a. 

Supervisor  ratings 

range 

b. 

Performance  Test 

Scores 

range 

c. 

Peer  ratings 

ran^ 

d. 

Promotion  record 

yes 

e. 

Safety  record 

range 

X = 
X = 
X - 


m 

X 

X 

X 

!< 

X 

X 

X 

X 

X 

X 

X 


X 

X 

X 


FIGURE  V.6:  Sumnary  Sheet  of  Data 
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One  output  of  this  block  will  be  the  completely  filled  In  data  spread 
sheets  on  students.  These  will  go  to  the  next  block,  V.2:  EXTERNAL 
EVALUATION.  They  should  be  retained  permanently  or  until  such  substantial 
changes  have  been  made  In  the  course  that  making  conparlsons  with  the 
original  course  Is  no  longer  possible.  By  that  time,  all  of  the  scores 
should  have  been  tabulated  and  Included  In  the  data  analyses  to  be  reported 
either  on  cards,  tape,  or  In  suitmari'  form.  Follow-up  data  on  eadh  student 
also  should  be  recorded  on  this  form. 

Block  III. 5:  VALIDATE  INSTRUCTION,  provided  a procedure  to  use  for 
deciding  whether  Instruction  was  adequate.  Generally,  these  same  procedures 
should  be  followed  for  the  entire  course  in  order  to  see  whether  the  course 
1s  Improving  and  where  additional  work  needs  to  be  done.  By  adding  the 
collected  time  data  and  the  additional  performance  data  obtained  from  each 
cycle  or  group  of  students  going  through  the  program,  gradually  a consider- 
able amount  of  evidence  will  be  gathered.  This  evidence  will  either  con- 
firm the  internal  quality  of  the  course,  or  point  out  areas  in  which 
revisions  may  be  required. 

When  Is  the  course  good  enough?  This  is  one  of  the  more  difficult 
questions  that  the  internal  evaluator  will  have  to  deal  with  on  a daily 
basis.  It  is  the  same  type  question  as:  "How  much  safety  is  enough?" 

The  ideal  situation  is  that  there  be  no  accidents  and  that  there  be 
no  school  failures.  Up  to  now,  and  probably  for  a while  in  the  future, 
not  enough  is  known  to  prevent  all  accidents  nor  to  design  "Zero  Defects" 
instruction.  What  can  be  said  Is  that  the  goal  is  zero  defects  in  instruc- 
tion, consistent  with  time  and  resource  availability. 
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Block  1.2:  SELECT  TASKS/FUNCTIONS  provided  a list  of  tasks  for  which 
training  was  required.  Since  this  task  list  was  very  carefully  screened 
for  unnecessary,  outdated,  and  nice-to-know  information,  It  Is  reasonable 
to  say  that  students  should  reach  an  acceptable  criterion  on  all  tasks 
selected  for  training.  That  is,  no  student  may  go  to  the  fleet  or  to  duty 
without  having  passed  all  of  the  internal  performance  measures.  This  is 
a good  rule  and  should  be  applied  consistently  1n  training.  There  will, 
of  course,  be  exceptions.  For  example,  if  a unit  o.’  fleet  is  known  to 
have  a FOJT  program  which  Is  Intended  to  mc  '**  trainees  rapidly  from  their 
school  performance  to  acceptable  command  performance,  there  might  be 
enough  overlap  In  the  two  areas  to  permit  some  leeway  in  deciding  when 
students  were  ready  to  leave. 

But,  if  there  is  no  training  program  in  the  receiving  command  and  the 
student  will  have  to  perform  well  the  day  he  arrives,  the  standards  set  in 
the  school  must  be  high  and  consistent,  and  no  one  should  leave  until  he 
has  mastered  the  objectives.  However,  all  of  this  must  be  thought  about 
in  the  context  of  local  command  requirements  and  the  availability  of  a 
particular  DOS  to  the  service.  Sometimes  more  people,  even  though  only 
partially  trained,  are  required  than  can  be  supplied.  This  will  put  a 
great  strain  on  the  training  program.  However,  if  training  has  been 

( 

I properly  designed,  one  of  the  features  it  could  have  is  that  of  rapidly 

I increasing  or  decreasing  the  number  of  graduates  without  requiring  a long 

lead  time.  To  establish  across-the-board  criteria  for  all  schools  under 
all  conditions  is  not  reasonable  and  may  not  even  be  possible. 

It  is  not  the  internal  evaluator's  job  to  decide  which  students  should 
pass  and  which  should  remain  for  additional  training.  His  job  is  to  try 
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to  find  out  If  the  instruction  has  net  the  requirements  expressed  In  the 
objectives,  tests,  and  other  evaluative  procedures.  This  Is  a very  diffi- 
cult distinction  to  make  since  most  past  experience  has  been  concerned 
with  the  making  of  decisions  about  students. 

In  many  courses  prepared  in  accordance  with  the  principles  of  ISO, 
students  will  have  the  opportunity  of  being  retested  several  times  on 
each  of  the  ma.jor  milestones  of  the  course,  at  least,  until  the  local 
training  installation  has  gained  enough  experience  to  know  whether  there 
is  a continued  payoff  for  tt/is  retesting  procedure. 

2.3.2  Display  and  Interpretation  of  Perfomance  Data 

A data  sheet  similar  to  the  one. In  Figure  V.7  should  be  prepared  for 
each  25,  50,  or  100  students  who  go  through  the  course.  The  example  data 
are  assumed  to  have  been  gathered  from  a course  evaluation  in  which  the 
following  procedures  were  used: 

1.  Students  were  atinitted  to  the  program  without  regard  to 
whether  they  met  the  prerequisites. 

2.  All  students  worked  through  the  course  in  the  same  order 
at  individual  rates. 

3.  Students  proceeded  through  the  course  regardless  of  the 
scores  they  made  on  entry,  pre-  and  posttests.  For  this 
trial  no  students  were  required  to  repeat  any  lessons. 

Those  who  failed  the  JPM  were  required  to  restudy  all  or 
parts  of  the  course  but  those  data  are  not  reported  here. 

4.  Units  end  lessons  within  units  were  roughly  sequential,  that 
is,  students  mastery  of  material  in  Unit  I lesson  3 was 
partly  dependent  on  their  mastery  of  material  in  lessons  I 
and  2 of  the  unit. 
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5.  The  entry  test  contained  Items  which  measured  skills  and 
knowledge  needed  by  students  to  learn  the  course  content 
but  which  was  not  taught  In  the  course.  For  example,  the 
course  might  involve  the  solution  of  percent  and  rate 
problems.  The  entry  skills  test  would  therefore  contain 
arithmetic  Items  which  measure  these  necessary  skills. 

6.  The  pretests  contained  Items  from  each  posttest  In  either 
identical  or  simp!  1 tied  fonr(. 

7.  Scores  on  the  JPM  occurred  naturally  in  go/no-go  form  or 
they  were  reduced  to  go/no-go  categories  using  guidelines 
provided  in  Block  1.3. 

The  following  diagram  shows  the  relationships  between  the  various 
parts  of  the  course: 


43 


Placement  of  the  course  components  shows,  from  left  to  right,  the  order  1n 
which  the  data  was  gathered.  The  arrows  Indicate  relationships  between 
the  con^onents  that  are  of  Interest  to  the  evaluator. 

The  important  evaluation  questions  that  can  be  answered  by  the  data 
are  listed  below  and  then  discussed  in  turn. 

1.  Wiiat  are  optimal  cut-off  scores  for  the  CRTs  of  Unit  1 
and  Unit  2? 

2.  Are  the  prerequisites  valid  in  the  sense  that  they  predict 
success  and  failure  in  the  course? 

3.  Can  the  pretest  provide  information  useful  in  placing 
different  students  at  different  levels  of  the  course? 

In  order  to  answer  the  first  question,  "What  are  optimal  cut-off  scores 
for  the  CRTs  1n  units  1 and  2?",  one  must  look  at  the  relationship  between 
the  last  unit  (in  this  case,  unit  2)  and  the  JPM.  Do  this  by  breaking 
the  unit  2 CRT  scores  into  two  or  more  categories.  Then,  using  the  data 
from  Figure  V.7,  construct  a table  which  shows  how  many  people  in  each 
CRT  category  passed  or  failed  the  JPM.  The  number  of  categories  chosen 
is  arbitrary  except  that  the  more  categories  used  the  more  students 
needed.  The  table  shown  in  Figure  V.8  has  four  unit  2-CRT  categories, 
and  within  each  category  it  shows  how  many  students  pass  or  fail 
the  JPM.  It  is  apparent  that  the  minimum  acceptable  score  for  the  unit 
CRT  should  be  either  30  or  25.  This  Is  because  93%  of  the  students  who 
attain  30  or  better  and  70%  of  those  who  get  25  or  better  pass  the  JPM. 
Which  is  optimal  depends  on  the  nature  of  the  JPM.  If  it  is  an  indi- 
vidually administered  costly  device,  then  the  higher  cutoff  score  would 
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be  chosen.  If  it  Is  easily  oJniitilstered  and  readily  available,  then 
the  lower  score  could  be  used.  Gf  course,  other  considerations  might 
also  enter  into  the  decision. 


JPM 


Below  19 

20-24 

25-29 

30-34 

Total 

N % 

N % 

N % 

N % 

Pass 

2 22 

6 33 

23  70 

37  93 

68 

Fail 

7 78 

12  66 

10  30 

3 7 

32 

Total 

9 100 

18  100 

33  100 

40  100 

100 

FIGURE  V.8:  Numbers  and  Percentages  of  Students 
in  Each  Unit  2 - CRT  Score  Category 
who  passed  or  failed  the  course  JPM. 


One  limitation  of  the  procedure  used  in  the  example  in  Figure  V.8 
is  that  no  clear  information  is  provided  concerning  which  lessons  need 
to  be  restudied  by  students  who  fail  to  reach  the  cutcff  score.  Student 
06  in  Figure  V.7  has  lesson  scores  in  Unit  2 of  7/10,  5/10,  6/8  and  6/6. 
Should  he  re- take  the  first  three  lessons  or  only  tne  second  one? 

Obviously  the  answer  could  be  provided  only  by  determining  cutoff  scores 
for  each  lesson.  If  the  lessons  are  fairly  short  and  if  they  appear  to  be 
of  equal  importance  the  overall  unit  score  should  be  used  because  it  is 
probably  more  reliable  (consistent)  than  the  individual  lesson  scores. 

If  it  was  necessary  to  determine  cutoff  scores  for  each  lesson,  then 
the  cutoff  score  for  lesson  4 should  be  dependent  on  having  mastered 


the  nuiterlal  in  lesson  3.  Otherwise  the  cutoff  score  for  lesson  3 should 
also  be  determined  with  the  IPM.  These  principles  also  apply  to  the  determ- 
ination of  cutoff  scores  f r lessons  2 and  1. 

The  cutoff  score  for  Unit  1 is  established  according  to  the  same  pro- 
cedures described  above  except  that  the  Unit  2 cutoff  score  is  used  instead 
of  the  JPM  since  we  are  assiming  that  mastery  of  Unit  2 is  depenctent  on 
mastery  of  Unit  1.  Figure  V.9  shows  the  kind  of  data  that  can  be  expected. 


Unit  2 

CRT  Pass  (25  & 


Fail  (below 
(25) 


Below  13 

14 

- 18 

19 

- 23 

24 

- 28 

Total 

N % 

N 

% 

N 

% 

N 

X 

10  50 

5 

50 

29 

78 

29 

88 

73 

10  50 

5 

50 

8 

22 

4 

12 

27 

20  100 

10 

100 

37 

100 

33 

100 

100 

FIGURE  V.9:  Ncmwers  and  Percentages  of  Students  in  Each 
Unit  1 - CRT  Score  Category  who  Passed  or 
Failed  the  Unit  2 - CRT 


Again  there  are  two  choices  tor  a cutoff  score  19  and  24.  Which  of  them 
is  chosen  nay  be  determined  by  other  factors. 

The  answers  to  the  other  two  evaluation  questions  can  be  arrived  at 
in  a similar  fashion.  To  answer  the  question,  "Are  the  prerequisites 
valid?",  construct  a table  similar  to  those  in-Figures  B and  9 for  each 
prerequisite.  Tables  for  this  question  aie  likely  to  yield  less  clear- 
cut  results  than  those  in  the  examples.  Suppose  you  had  50  E5s 
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and  50  E8s  in  your  sanple  and  38  (76S)  of  the  E5s  passed  the  JPM  after  the 
course,  but  only  ^ {&>%)  of  the  E8s  passed.  Poes  this  result  mean  that 
all  E5s  are  more  likely  to  be  successful  than  all  E8s,  or  could  the  16% 
difference  in  success  rate  be  due  to  chance?  That  is,  if  other  samples 
of  E5s  and  E8s  were  used  would  the  E5s  always  have  a higher  success  rate? 

In  order  to  answer  this  question  you  need  to  consult  someone  who  has 
training  in  statistics  if  you  do  not  have  such  capabilities  yourself. 

The  optimal  use  of  a pretest  in  placing  students  within  the  course 
depends  on  the  number  of  entry  points  and  the  length  of  the  CRTs  asso- 
ciated with  them.  In  the  current  example  there  are  only  three  decisions 
that  can  be  made  with  the  pretest:  place  the  student  in  Unit  I,  place 
him  in  Unit  II,  or  administer  the  JPM  to  him  with  no  further  training. 

The  pretest  could  have  two  subtests,  one  for  each  of  the  two  regular 
units.  A student  who  passed  both  would  go  directly  to  the  JPM.  In  the 
present  example  the  subtests  for  Units  1 and  2 could  be  identical  with  or 
alternate  forms  of  the  unit  posttest  CRTs.  Passing  scores  for  them  would 
have  been  established  in  the  procedures  previously  described  and  no  further 
work  would  have  to  be  done  on  them.  If  there  were  many  entry  points  into 
the  program  or  if  the  CRTs  were  very  time  consuming  it  might  be  reasonable 
to  construct  short  tests  that  could  predict  performance  on  the  CRTs.  In 
that  case,  of  course,  each  subtest  would  have  to  be  paired  with  its  unit 
CRT  and  a table  constructed  to  determine  whether  it  did  in  fact  predict 
properly  and  to  determine  its  cutoff  score. 

2.3.3.  Sone  Additional  Evaluation  Designs 

In  all  discussions  of  validation  and  evaluation  up  to  this  point,  you 
have  been  advised  to  use  what  is  called  a one-group  pretest-posttest  design. 
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In  using  it,  stU(tents  are  located  who  have  low  scores  on  a pretest, 
training  is  administered,  and  then  a posttest  is  given  to  them.  If 
scores  on  the  posttest  are  substantially  higher  than  those  of  the 
pretest  (and/or  meet  seme  performance  standard)  you  conclude  that  the 
training  was  reponsible  for  the  change.  Tills  conclusion  is  generally 
reasonable  when  the  period  of  instruction  is  short  and  the  tests  are 
reliable  JPMs.  There  may  be  situations  where  such  a conclusion  is  open 
to  doubt.  It  may  be  that  simply  taking  the  pretest  provided  sufficient 
practice  for  students  to  in^rove  their  scores  on  the  posttest.  Or  it 
may  be  that  the  pre-posttests  are  not  reliable. 

Consider  an  extreme  example  in  which  the  pretest  asked  students  to 
list  the  outcome  of  a series  of  ten  coin  tosses.  In  a relatively  large 
group  tlie  average  score  will  be  five,  but  some  students  will  have  a 
scot*e  of  zero.  If  the  test  is  administered  a second  time  they  are  most  like 
ly  to  have  an  average  score  of  five  since  there  is  no  reason  for  them  to  be 
"unlucky"  twice.  If  one  wins  unaware  of  the  value  of  the  tests  and  be- 
lieved that  some  instruction  occured  between  them,  he  might  conclude  that 
the  pre-post  differences  were  due  tn  instruction. 

j ■ 

All  tests  that  are  not  perfectly  reliable  contain  chance  elements  that 
tend  to  make  initially  low-scoring  groups  have  higher  scores  the  second 
time  the  test  is  taken.  These  same  chance  elements  tend  to  make  initially 
high  group  scores  lower  the  second  time;  hoxever,  these  individuals  usually 
are  eliminated  from  the  subject  satiric.  Therefore,  you  are  rarely  if  ever 
led  to  tile  mistaken  conclusion  that  the  instruction  made  the  subjects  less 
competent  than  before. 
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There  are  a number  of  other  reasons  that  might  cause  doubt  whether 
the  inct'ease  In  posttest  scores  was  due  to  the  Instruction.  In  order  to 
eliminate  all  of  these  possibilities,  use  a design  that  employs  a control 
group.  That  Is,  have  one  group  (the  treatment  group)  take  the  pretest, 
undergo  Instruction,  and  take  the  posttest.  The  control  group  only  takes  tlie 
pre  and  posttests  at  the  same  times  the  first  group  takes  them.  Ideally 
you  should  assign  students  randomly  to  the  groups,  with  all  of  them  being 
low  on  the  pretest.  If  you  then  observe  that  the  treatnent  group  has 
higher  scores  on  the  posttest  than  the  control  group,  you  can  conclude  that 
the  Instruction  provided  was  responsible  for  the  Increase. 

Sometimes  the  one-group-pretest  posUest  design  Is  simply  Incapable  of 
answering  the  evaluation  questions  you  want  to  ask.  Suppose  you  have  design- 
ed a course  In  land  navigation  which  contains  an  entry  test,  a series  of  AV 
lessons,  an  audio  cassette  exercise,  a simulation  game,  a field  exercise  and 
a JPH.  The  evaluation  question  might  be,  are  all  elements  of  the  course 
necessary  to  produce  satisfactory  performance  on  the  JPM?  One  relatively 
simple  design  that  will  provide  at  least  a partia'*  answer  to  the  questions 
follows:  After  giving  the  entry  test,  assign  (rai.domly  If  possible)  students 
to  one  of  the  training  conditions  given  below: 

1.  AV  lessons,  JPH 

2.  AV  lessons.  Audio  cassette,  JPM 

3.  AV  lessons,  audio  cassette,  simulation  game,  JPM 

4.  AV  lessons,  audio  cassette,  simulation  game,  field 

exercise,  JPM 

Si4>pose  you  found  the  following  percentages  of  passes  on  the  JPM  in  the 
four  groups,  64,  77,  79,  91.  You  would  probably  conclude  that  the  simu- 
lation game  added  little  to  tiie  course  and  that  it  could  be  eliminated  with- 
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out  loss  of  effectiveness.  The  design  above  is  based  on  the  assumption 
that  Instruction  in  the  course  is  cummulative  so  that  it  would  not  make 
sense  to  ask,  for  example,  whether  the  audio  cassette  and  field  exercise 
combination  was  as  effective  as  any  other  combination.  If  this  assumption 
is  not  reasonable  and  you  wanted  to  find  out  the  best  combinations  and 
thie  four  elements  you  would  need  16  groups  (2x2x2x2). 

Evaluation  designs  can  become  very  complex,  and  in  addition,  often 
require  sophisticated  statistical  techniques  for  proper  interpretation 
of  their  results.  If  you  beccxne  convinced  that,  for  a particular  evalua- 
tion situation,  the  one-group-pretest  posttest  design  may  give  misleading 
results  or  is  not  capable  of  answering  the  questions  you  want  tc  ask,  you 
may  need  to  employ  the  services  of  an  evaluation  or  research  specialist 
or  get  additional  training  in  that  area. 

3.0  OUTPUTS 

The  outputs  of  this  block  should  consist  of  the  following: 

3.1  Products 

An  internal  evaluation  report  (INER).  This  report  is  a summary 
statement  of  the  internal  evaluation  findings,  their  interpretation, 
and  specific  recotimendations  for  revision  of  the  instruction  (see 
Table  V.3), 

3.2  Other  Documentation 

Supporting  information  including,  as  a minimum,  the  following; 

1.  Complete  student  records  containing  all  pertinent  information 
on  all  students  who  entered  the  instruction  being  evaluated. 


: SmAKi  OF  If<TERNAL  EVALUATION  REPORT 
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2.  The  degree  to  which  the  ISO  process  was  followed,  making 
certain  that  any  exceptions  are  noted  and  explained.  The 
evaluation  of  the  outputs  on  a block  by  block  basis. 

3.  The  developmental  time  and  resources  used  to  develop  the 
course.  Including  unique  and  unusual  costs  or  savings 
brought  about  by  the  specific  procedures  followed, 

4.  A profile  of  the  entry  characteristics  of  the  students,  in 
terms  of  their  personal  characteristics,  test  scores, 
selection  criteria,  and  other  personal  history;  and,  the 
specific  performance  requirements  that  had  to  be  met  by 
them  prior  to  beginning  the  instruction  In  the  main  part 
of  the  course.  A discussion  of  the  planned  vs.  actual 
entry  behaviors  of  the  students. 

5.  A completely  detailed,  and  summarized,  presentation  of 
student  performance  data,  organized  by  major  objective, 
or  tasks.  Also  Included  should  be  the  summaries  of 
student  questionnaires  and  other  data  provided  by  students. 

6.  A summary  of  the  major  Inputs  from  the  instructors,  including 
any  analyses  i..ade  of  the  critical  incident  reports  or  the 
questionnaires  completed  by  the  instructional  and  testing 
staffs . 

It  Is  necessary  to  decide  the  regularity  or  frequency  of  the  internal 


evaluation  report  for  any  course;  that  Is,  whether  the  reporting  period 
should  be  based  on  numbers  of  students,  or  should  it  be  sent  at  regular 
times.  If  it  Is  to  be  used  principally  for  reporting  results,  it  will 


probably  be  necessai^  to  suboiit  it  regularly.  If  it  is  principally  to 
be  used  for  decisions  about  revising  the  course.  i«.  can  be  submitted 
followiiig  the  completion  of  the  course  by  a significt  t nuiitter  of  stu- 
dents. Earlier  reports  probably  should  be  based  on  fewer  students  then 
later  reports.  Ai\y  serious  problems  probably  will  occur  early  and 
shouV'  be  given  immediate  attention.  Less  serious  problcns  should 
be  well -substantiated  by  a larger  number  of  students  before  revision 
1r  begun. 


EXAMPLE 


These  data  suggest  that: 

1.  Neither  age,  length  of  service  nor  years  of  military 
schooling  relate  in  any  systematic  way  to  performance 
in  school  or  on  the  job.  Therefore,  these  administra- 
tive requirements  are  not  relevant  to  training  and  it 
is  recoitinended  that  they  be  dropped. 

2.  Units  2 and  5 reflect  much  lower  average  scores  than 
the  other  units.  Additional  Instruction,  or  additional 
time  for  i;he  same  instruction,  will  be  considered. 

3.  Two  of  the  three  individuals  who  failed  the  entry  test 
performed  satisfactf'rily  in  school  and  on  the  job, 
even  though  they  had  no  remedial  training.  The  entry 
test  is  therefore  too  difficult  and  should  be  modified. 


APPENDIX  A 

EXAMPLES  OF  DATA  FOR»C  AND  QUESTIONNAIRES 
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EXAMPLES  OF  DATA  FORMS  AND  QUESTIONNAIRES 

Pi recti ons:  Observe  the  behavior  of  the  learner  In  the  classroom  and  keep 

notes  and  frequenpy  counts  of  what  happens  In  each  of  the  following  cate- 
gories. When  the  observation  period  has  been  completed,  use  your  notes 
and  frequency  counts  to  complete  this  form.  Upon  cotipletion  submit  to 
the  Project  Officer  for  Internal  evaluation.  This  Project  Officer  may 
ask  for  your  assistance  and  for  clarification  during  the  Analysis  phase 
of  this  evaluation. 


A. 

Evaluation 

NAME 

OFFICE 

TEL. 

B. 

Length  of  time  for  observation: 

C. 

Location  of  observation: 

D. 

Title  of  Learning  Block: 

E. 

Did  this  observation  cover  the  entire  block? 

F. 

If  not,  which  part? 

G. 

Instructor: 

"nShE 

OFFICE 

TEL. 

H. 

Time  of  day:  . I . 

Number  of  students; 

SECTION  I.  - THE  LEARNER 


(answer  In  terms  of  per  hour)  (If  not  observed  enter  N/0,  if  not 

applicable,  enter  NM) 

1.  How  frequently  did  student  questions  indicate  a lack  of  knowledge 

of  tha  objective  required? 


Never 


Seldom  Often  Frequently  Constantly 
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How  frequently  did  student  quest ions/connents  indicatu  they  were 
aware  of  the  objective  but  didn't  understand  It? 


Never  Seldom  Often  Frequently  Constantly 

How  frequently  did  student  questlons/comments  indicate  that  they 
understood  the  objective  but  didn't  see  where  It  fit  Into  the  job 
they  were  learning? 


Never  Seldom  Often  Frequently  Constantly 

How  frequently  did  student  questlons/comments  Indicate  that  they 
did  not  understand  how  much  assistance  they  could  expect  to  receive 
when  performing  this  part  of  their  job? 


Never  Seldom  Often  Frequently  Constantly 

How  frequently  did  student  questlons/comments  Indicate  that  they 
did  not  know  which  tools  they  would  be  allowed  to  use  on  the  job? 


Never  SeTHoin  Often  frequently  Constantly 

How  frequently  did  student  questlons/comments  Indicate  that  they 
did  not  know  what  materials  they  would  receive  to  do  the  job? 


Never  Seldom  Often  Frequently  Constantly 

How  frequently  did  student  questions/coirments  Indicate  that  they 
did  not  know  which  references  they  would  receive  on  the  job? 


Never  Seldom  Often  Frequently  Constantly 

How  many  times  while  performing  did  learners  fail  to  use  any  of 
the  materlals/tools/references/etc.  they  could  use  on  the  job? 


Never  Seldom  Often  Frequently  Constantly 

How  many  times  while  performing  did  the  learners  use  a material /tool/ 
reference/etc.  that  they  would  NOT  receive  on  the  job? 


Never  Seldom 


Often  Frequently  Constantly 
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10.  frequently  dlw  student  questions  indicate  that  they  were  confused 
as  to  where  a classroom  required  behavior  fit  into  the  Job-at-large? 


Never  Seldom'  Often  Frequently  Constantly 

11.  How  frequently  did  student  questions  about  a test  Indicate  that  they 
did  not  relate  the  classroom  activity  with  the  job  performance 
requirement  of  the  test? 


Never  Seldom  Often  frequently  Constantly 

12.  What  percent  of  student  failure,  on  a test,  indicated  a lack  of 
ability  to  transfer  classroom  learning  to  job-oriented  requirements 
in  the  test? 


Never  Seldom  Often  Frequently  Constantly 

13.  What  percent  of  the  class  had  a great  difficulty  in  starting  the  test? 


Never  Seldom  Often  Frequently  Constantly 

14.  How  frequently  did  the  learner  require  tools,  equipment,  references, 
etc.,  that  he  had  used  in  the  classroom  if  they  were  not  supplied 
with  the  test? 


Never  Seldom 

Often 

Frequently 

Constantly 

[:■  15. 

t' 

1 

What  percent  of  the 
time? 

learners 

finished  the 

test  within  the  allotted 

1-10« 

11-50% 

51-80% 

81-100% 

i 16. 

f. 

What  percent  of  the 

learners 

performed  below  the  desired  standard? 

OX  1-lOX 

“11-50% 

51-80% 

81-100% 

17.  How  frequently  did  learners  questions  indicate  that  they  did  not  know 
what  standard  would  be  used  to  enforce  their  behavior  in  the  test? 


Never  Seldom  Often  Frequently  Constantly 

18.  How  frequently  did  the  learners  questions/answers  indicate  that  they 
did  not  have  the  necessary  requisite  skills  and  knowledges  to  begin 
the  learning  of  new  behavior? 


Never  Seldom  Often  Frequently  Constantly 


I 


I 
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19.  Warn  frequently  did  the  learner's  qi^stlons/connents  Indicate  that 
could  not  regroup  his  entry  level  skills  to  perform  In  the  nmvly 
required  behavior? 


Hever~  Seldoa  Often  Frequently  Constantly 

20.  Huh  frequently  did  th«  learner's  questlons/cc^ments  Indicate  that 
he  could  not  transfer  entry  level,  or  newly  acquired  skills  to  the 
performance  of  required  behaviors? 


liiver*’  Seldom  Often  Frequently  Constantly 

21.  What  percent  of  the  students  failed  to  complete  the  requirements  of 
the  learning  event  In  the  time  allotted? 


"FW  11-50X  51-OOX  81-lOOS 

22.  How  frequently  did  student  questions/comments  indicate  that  he  did 
not  recognize  the  relationship  between  learning  events? 


Never  SeT3Sn  Often  Frequently  Constantly 

23.  What  percent  of  the  students  conpleted  the  requirements  of  the  learning 
event  with  too  much  time  left  over? 


“OF”  1-lOX  11-50*  51-®*  Ol-lOO* 

24.  How  frequently  did  student  questions/coirments  indicate  dissatisfaction 
with  the  learning  event? 


Never  Seldom  Often  Frequently  Constantly 

25.  How  frequently  did  student  questions/comments  Indicate  a lack  of 
recognition  of  how  the  learning  event  related  to  job  performance 
requirements? 


Never  Seldom  Often  Frequently  Constantly 

26.  How  frequently  did  student  questions/comments  indicate  that  they  ware 
not  rea^  to  develop  a new  behavior? 


Never  Seldom  Often  Frequently  Constantly 


i 

{ 
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27.  Kow  frequently  did  student  progress  Indicate  that  they  did  not  possess 
required  entry  level  behaviors? 


Never  Se1(k»i  Often  Frequently  Constantly 

28.  tow  freqi^ntly  did  student  questlons/cMments  Indicate  that  the 

delivery  system  being  en^jloyed  did  not  make  provision  for  Individual 
RATES  of  learning? 


Never  Seldom’  Often  Frequently  Constantly 

29.  How  frequently  did  student  questlons/conments  Indicate  that  the  delivery 
system  being  employed  did  not  make  provision  for  the  entry  level 
differences  of  the  learners? 


Never  Seliom  Often  Frequently  Constantly 

30.  How  frequently  did  student  questions /comments  Indicate  that  the  delivery 
system  being  employed  did  not  make  provision  for  the  differences  In  the 
learner's  quality  of  learning? 


Never  Seldom  Often  Frequently  Constantly 

31.  How  frequently  did  student  questlons/cooinents  Indicate  that  the  delivery 
systen  being  employed  did  not  make  provision  for  the  learner  differences 
in  the  quantity  they  could  learn? 


Never  Seldom  Often  Frequently  Constantly 

32.  How  frequently  did  student  quest! ons/comments  Indicate  that  the  delivery 
system  being  employed  did  not  make  provision  for  the  learner  differences 
In  being  able  to  deal  with  the  Intensity  of  the  learning  experience? 


Never  Seldom  Often  Fi^qwntTy  Constantly 

33.  What  percent  of  the  learners  were  deterred  from  fulfilling  the  learning 
objective  because  of  the  complexity  of  the  delivery  system? 


Never  Seldom  Often  Frequently  Constantly 

34.  Mow  frequently  did  student  questions/comments  indicate  that  they  did 
not  understand  the  relationship  between  the  delivery  system  being 
employed  and  the  job  performance  requirement  being  developed? 


Never  Seldom  Often  Frequently  Constantly 
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35.  How  frequently  did  student  questlons/comnents  Indicate  that  the  leaner 
Is  not  receivino  the  necessary  relearning  from  the  delivery  system? 


Hewer  Seldom  Often  Frequently  Constantly 

36.  How  frequently  did  students  questlons/comnents  indicate  that  they 
found  the  learning  materials  difficult  to  use? 


Never  Seldom  Often  Frequently  Cons  tan ‘Ty’ 

37.  How  frequently  did  students  questions/conments  indicate  that  the 
learning  materials  were  hindering  instead  of  helping  learning? 


Never  Sel dom  Often  Frequently  Constantly 

38.  How  frequently  did  students  questions/coinments  Indicate  that  the 
learning  materials  failed  to  provide  for  the  learner  differtnees 
in  Rate  of  Learning? 


Never  Seldom  Often  Frequently  Constantly 

39.  How  frequently  did  students'  questions/conments  indicate  that  the 
learning  materials  failed  to  provide  for  the  learner's  differences 
in  level  of  learning? 


Never  Seldom  Often  frequently  Constantly 

40.  How  frequently  did  student  questions/comments  indicate  that  the 
learning  materials  failed  to  provide  for  the  differences  in  the 
learners'  quality  of  learning? 


liever  Seldom  Often  Frequently  Constantly 

41,  How  frequently  did  student  questions/conments  indicate  that  the 
learning  materials  failed  to  provide  for  the  differences  in  the 
learner's  quantity  of  learning? 


I 

i 

s 

I 

i 

i 

! 


Never  Seldom  Often  Frequently  Constantly 

42.  How  frequently  did  student  questions/conments  intricate  that  the 
learning  materials  failed  to  provide  for  the  differences  in  the 
learner's  ability  to  cope  with  the  intensity  of  the  learning 
experience? 


Often 


Never 


Sel dom 


Frequently  Constantly 
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43.  Hom  frequently  did  student  questfons/co^ients  Indicate  that  they  could 
not  Identify  the  learning  material  with  the  Job  performarKe  requirement 
being  developed? 


Never  Seldom  ffften  Frequently  Constantly 

44.  What  percent  of  the  time  were  the  learners  Inactive  or  passive 
participants  In  the  learning  situation? 


w 1-10%  n-503:  5i-®r 

45.  How  frequently  did  student  questlons/coimtents  Indicate  that  their 
behavior  had  not  been  changed  to  match  the  specifications  of  the  job 
performance  requirements? 


Never  Seldom  Often  Frequently  Constantly 

46.  How  freqi^ntly  did  student  questlons/conments  Indicate  that  they  had 
not  received  sufficient  relearning. 


Never  Seldom  Often  Frequently  Constantly 


INSTRUCTORS 

1.  How  freqittntly  did  the  Instructor  make  attenpts  to  Insure  that  the 
learner  knew  what  the  c^Jectlve  (behavior  being  developed)  was? 


fJ^er  Seldom  "Sften  Frequently  Constantly 

2.  How  frequently  did  the  instructor  denwnstrate  the  job  performance 
requirement  that  was  to  be  developed  by  the  learner? 


Jtever  SiTSm  Often  Frequently  Constantly 

3.  How  frequently  did  the  instructor  describe  the  tools,  assistance, 

materials,  referencer,  etc.,  that  the  learner  would  receive  to  perform 
the  desired  behavior? 


Never  Sel<k)m  Often  Frequently  Constantly 

4.  How  frequently  did  the  instructor  cteironstrate  the  use  of  the  tools, 
materials,  references,  etc.,  that  the  learner  would  receive  to 
perform  the  <tesi red  behavior? 


Never  Seldom  Often  Frequently  Constantly 
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I ALL  DOCUMENTATIt'N 
INPUT  FOR  ALL  BLOCKS  IN 
PHASES  I.  n.dmira. 


DEVELOP  INTERNAL 
EVALUATION  PLAN 


DEVELOP  PROGRESS 
EVALUATION  PLAN 


CONDUCT  INTERNAL 
EVALUATION 

2.2 

; 

p 



1 

INTERPRET  DATA  AND 
MAKE  RECOMMENDATIONS 
2.3 

DEVELOP  PROCESS 

EVALUATION  PLAN 

2.1.2 

'BLOCK^ 
V .2  jmJ 
.3 


DEVELOP  PERFORMANCE 
EVALUATION  PLAN 

2 1.3 


DEVELOP  PLAN  FOR 

COLLECTING  INFORMATION 

FROM  STUDENTS 

2.1.4 

DEVELOP  PLAN  FOR 
COLLECTING  INFOl  MATION 
FROM  INSTRUCTORS  2 ‘ 0 


FIGURE  Y.1:  Flowcharto'  TIocklT.I.  CONDUCT  INTERNAL  EVALUATION 


! 

i 
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OVERVIEW 


After  the  internal  evaluation  ha''  ueen  completed,  one  major  question 
about  the  training  program  remains  unanswered.  This  is  whether  students 
who  have  completed  the  program  and  have  been  placed  on  the  job,  can  do  the 
job  fo.”  which  they  were  trained.  External  evaluation  is  accomplished  to 
answer  that  question  and,  if  the  answer  -is  "no,"  to  find  out  why. 
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CONDUCT  EXTERNAL  EVALUATION 
I.O  IN'mODilCTION 


The  primary  purpose  for  conducting  external  evaluation  Is  to 
find  out  whether  students  who  successfully  complete  training  can  do, 
to  the  est'blished  standards,  the  job  for  which  they  were  trained.  A 
second  purpose  of  external  evaluation  is  to  Insure  that  the  job  is 
still  the  same  as  when  It  was  job  analyzed.  In  a sense  this  revalidates 
the  original  task  list. 

External  evaluation  is  different  from  internal  evaluation  in  two 
major  ways.  First,  while  Internal  evaluation  is  conducted  before  and 
during  the  actual  instruction,  external  evaluation  is  conducted  after 
the  students  have  completed  the  instruction  and  have  been  assigned  to 
the  job.  Second,  internal  evaluation  is  concerned  with  whether  the 
learning  objectives  developed  in  Block  11". 1 are  mastered  by  the  students. 
Whether  the  learning  objectives  are  mastered  is  determined  by  student 
performance  on  the  tests  developed  in  Block  1 1. 2.  But  this  does  not 
ensure  that  Block  I.l  through  Block  1.5,  the  analysis  phase  of  the  ISD 
Model,  were  properly  carried  out.  Neither  does  it  ensure  that  the 
job  tequirements  are  still  basically  the  same  as  when  the  program  de- 
velojxnent  was  begun.  Block  V.l  detailed  other  important  interests  in 
internal  evaluation  and  showed  how  it  relates  to  external  evaluation. 

The  results  of  a properly  planned  and  carried  out  external  evalu- 
ation will  result  in  either: 

1.  Assurance  that  students  who  complete  the  course  will  be 
able  to  do  the  job  to  the  level  of  their  training,  or 

2.  Data  that  indicate  that  students  are  not  able  to  do  the 
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.lob  to  the  planned  standard,  and  on  which  of  the  Job 
tasks  they  are  falling. 

The  steps  in  planning  and  carrying  out  external  evaluation  are 
shown  in  Figure  V.IO,  the  fold-out  page  at  the  end  of  this  block. 


2.0  PROCEDURES 


2.1  Plan  Evaluation 


Before  beginning  the  external  evaluation 
some  important  decisions  must  be  made  as  to 
how  the  evaluation  will  be  conducted.  The 
following  determinations  must  be  made: 

t.  Who  will  i^rovide  data 

2.  What  data  are  required 

3.  When  external  evaluation  will 
take  place 

4.  How  the  data  will  be  gathered 


2.1.1  Determine  Who  Will  Provide  Data 


In  planning  the  external  evaluation,  cne  of  the  first  concerns  is 
the  question  of  who  will  provide  data.  In  general,  data  will  come 
from  five  sources: 

1.  Baseline  data  gathered  before  development  of  instruction 
was  started  (Sec  Block  II. 2.) 

2.  Graduates  being  evaluated.  These  are  graduates  who  are 
now  working  on  the  job  for  which  the  instruction  provided 
training 

3.  Supervisors  of  the  graduates  listed  above 
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4.  An  evaluation  team,  usually  made  up  of  cotnnand  and 
school  personnel 

5.  Records  of  students'  perforiiiance  during  the  Instruction 
In  addition  to  the  above,  check  out  some  of  the  graduates  who 

are  now  assigned  to  other  jobs.  Why  were  they  assigned  to  other 
jobs?  Were  they  first  assigned  to  the  job  for  which  they  were 
trained,  and  then  reassigned  to  a different  job  because  they  cou]d  not 
do  the  first  one? 


2.1.2  Determine  What  Data  are  Reoulred 


By  making  comparisons  between  the  baseline  data  and  data  gathered 
from  the  other  four  sources,  you  will  be  able  to  obtain  at  least  partial 
answers  to  questions  like  the  following: 

1.  Do  a greater  percentage  of  graduates  of  the  current  in- 
struction show  satisfactory  performance  on  the  JPMs  than 
graduates  of  older  courses  or  training  methods? 

2.  Do  graduates  of  the  current  instruction  require  less  on-the- 
job  training  than  graduates  of  other  courses  or  training 
methods? 

From  the  graduates  who  are  now  working  on  the  job,  get  answers  to 
such  questions  as: 

1.  How  well  are  you  able  to  perform  the  job? 

2.  How  much  and  what  kind  of  training  have  you  received  since 
you  arrived  on  the  job? 

3.  How  well  did  the  instruction  prepare  you  for  the  job? 

4.  What  portions  of  the  Instruction  were  relevant  to  your  job? 

5.  What  portions  of  the  instruction  were  irrelevant  to  your  job? 
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6.  In  your  Jcb,  how  often  do  you  use  the  skills  taught? 

7.  In  your  job,  what  tasks  have  given  you  the  most  difficulty? 

8.  In  your  Job,  for  which  tasks  do  you  feel  the  least 
adequately  prepared? 

9.  In  your  Job,  which  tasks  do  you  think  you  perform  the  best? 

10.  What  parts  of  the  Instruction  do  you  think  could  be  changed 

to  oetter  prepare  students  for  the  job? 

From  the  supervisors  of  graduates  who  are  now  working  on  the  job, 
get  answers  to  such  questions  as: 

1.  how  well  are  the  graduates  able  to  perform  on  the  job? 

2 How  do  these  graduates  compare  to  those  who  received  no 
training  or  were  trained  by  an  alternate  method? 

3.  How  much  and  what  kind  of  training  have  the  graduates  re- 
ceived since  arriving  on  the  job? 

4.  In  what  areas  were  the  graduates  the  most  adequately 
prepared? 

5.  In  what  areas  were  the  graduates  inadequately  prepared? 

6.  What  suggestions  would  you  make  for  improving  the  training 
program? 

7.  Has  the  graduate  had  accidents  or  been  reprimanded  for 
misuse  or  improper  operation  of  equlpi.ient? 

8.  Has  the  graduate  been  warned  or  commended  for  unusually 
good  or  bad  performances? 

9.  Has  the  graduate  been  recommended  for  promotion? 

From  the  evaluation  team,  get  answers  to  such  questions  as: 
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1.  How  well  did  graduates  score  on  the  job  performance  measures? 

2.  Which  JPMs  did  they  fail? 
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3.  What  is  the  relationship  between  the  JPMs  and  the  actual 
job  requirements? 

4.  Were  performance  tests  properly  acbnlnlstered  and  scored? 

5.  Does  the  command  perform  the  job  In  accordance  with 
regulations  or  approved  doctrine? 

6.  How  well  does  the  supervisor  know  the  job? 

7.  What  other  factors  should  be  considered  in  making  re- 
vision decisions? 

2.1.3  Oetennine  When  External  Evaluation  Will  Take  Place 

You  generally  will  not  want  to  contact  a graduate  or  the  graduate  s 
supervisor  until  the  graduate  has  been  on  the  job  for  at  least  30  days. 
This  is  because  it  probably  will  take  that  long  for  the  graduate  to 
get  some  feel  of  how  well  the  instruction  prepared  him  for  the  job, 
and  exactly  what  the  real  job  is  all  about.  Also,  it  probably  would 
take  that  long  for  the  supervisor  co  get  a clear  picture  of  how  well 
the  new  graduate  can  perform. 

Do  not  wait  more  than  three  months  before  contacting  the  graduate 
and  his  supervisor.  After  that  period,  the  graduate  will  be  hard 
pressed  to  remember  details  of  the  instruction.  He  likely  will  have 
a difficult  time  making  realistic  judgments  about  how  well  it  prepared 
him  for  the  job.  Also,  the  supervisor  likely  will  have  difficulty  in 
recalling  intervening  training  and  the  graduate's  initial  capabilities, 

2.1.4  Determine  How  the  Oato  Will  Be  Gathered 

The  primary  methods  of  collecting  external  evaluation  data  are: 

1.  Job  Performance  Measures 
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2.  Questionnaires 

3.  Personal  Interviews 

In  general,  you  will  send  questionnaires  to  as  many  graduates  and 
their  supervisors  as  possible,  and  c(»iiduct  personal  Interviews  with  a 
random  sample  of  yiaduates  and  their  supervisors  In  the  field.  The 
Job  performance  evaluation  approach  generally  Is  conducted  by  a team 
of  subject  matter/evaluatlon  experts  who  gather  data  from  actual  ob- 
servation of  the  gradual  on  the  job,  and  from  job  performance  measures 
Application  of  these  data  collection  methods  will  be  discussed  in 
greater  detail  later  In  this  block. 

2.2  Collect  Data 

The  actual  collection  of  external 
evaluation  data  is  a relatively  straight 
forward  group  of  procedures  once  a de- 
tailed plan  has  been  prepared.  These 
procedures  are: 

1.  Collect  baseline  data 

2.  Collect  job  performance 
evaluation  data 

3.  Collect  questionnaire  data 

4.  Collect  personal  interview  data 
Obtain  records  of  students'  performance  during  instruction 
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2.2.'  Collect  Baseline  Data 

In  Block  II. 2,  it  was  suggested  that 
you  gather  baseline  data  with  the  JPMs  in 
order  to  confirm  the  conclusions  that  the 
tasks  selected  for  training  were  not 
generally  being  adequately  performed  by 
job  Incumbents.  The  details  of  how  to 
conduct  such  a study  were  left  to  be 
discussed  in  this  block. 

A baseline  study  is  simply  a study  of  what  currently  exists.  It  can 
be  used  both  to  confirm  the  need  for  the  development  of  new  instruction, 
and  once  the  instruction  is  developed,  to  provide  data  for  the  external 
evaluation  of  the  new  instruction.  In  order  to  give  meaningful  results 
for  the  first  purpose,  a baseline  study  must  satisfy  two  basic  require- 
ments : 

1.  The  OPMs  (or  other  data  gathering  devices)  must  yield 
reliable  and  valid  information,  and 

2.  The  persons  to  whom  the  JPMs  are  administered  must  be 
representative  of  all  the  persons  (population)  to  whom 
the  results  will  apply. 

In  addition  to  these  requirements,  the  group  on  whom  data  is 
gathered  must  not  be  substantially  different  from  the  group  on  whom  the 
external  evaluation  data  will  be  gathered  if  the  data  are  to  be  used 
for  the  second  purpose.  If  the  procedures  outlined  in  Block  1.3  have 
been  followed,  it  is  likely  that  the  JPMs  developed  will  be  satisfactory 
for  the  baseline  study.  Selecting  a sample  that  is  truly  representative 


72 


the  regulations  in  effect  at  the  tiw!.  Job  performance  measures  that 
are  a part  of  the  regular  pro«K)tion  systen  will  not  normally  be  as 
complete  as  required  to  evaluate  school  performance;  therefore,  additional 
performance  measures  may  need  to  be  given.  This  will  all  depend  on 
the  job,  the  regulations,  and  the  availability  of  performance  measures. 

In  Block  1.3  of  the  ISO  Model,  it  was  stated  that  there  must  be 
some  kind  of  a JPM  developed  for  each  of  the  tasks  selected  for  training. 
If  this  was  followed,  there  will  be  valid  JPMs  to  use  in  the  field 
Setting.  If  this  was  not  followed,  external  evaluation  can  never  be 
more  accurate  than  the  interview  and  impressionistic  data  which  is  gath- 
ered in  that  manner.  That  is,  there  will  be  no  hard  data.,  no  direct 
measure,  of  the  performance  of  graduates.  Decisions  based  on  this 
latter  kind  of  data  are  far  riskier  than  decisions  made  on  the  basis  of 
validated  JPMs. 

Where  constraints  of  time,  equipment  and  facility  availability, 
cost,  and  safety  considerations  are  not  critical,  job  performance 
evaluation  consists  of  actual  observation  of  the  graduate  while  he 
does  his  job.  However,  since  in  many  cases  serious  constraints  will 
exist,  JPMs  developed  in  Block  1.3  will  have  to  be  evaluated  rather  than 
tb:  actual  job.  For  example,  suppose  a graduate  has  been  trained  to 
splint  a fractured  leg.  The  job  performance  evaluation  team  wishes  to 
know  how  well  he  can  perform  this  task  in  the  job  setting.  It  is  high- 
ly unlikely  that  someone  will  be  so  cooperative  as  to  wait  ur*-il  needed 
to  break  his  leg  so  that  the  graduate  can  be  observed  while  he  splints 
the  leg.  Actually,  one  would  have  to  watch  for  a long  time  to  observe 
graduates  performing  this  particular  task  under  real-world  conditions. 


This  was  one  of  the  reasons  for  developing  JPMs.  For  external 
evaluation,  where  constraints  make  it  impractical  to  evaluate  the 
actual  performance  of  all  tasks  for  which  the  graduates  were  trained, 
the  evaluation  can  then  be  based  on  the  JPH.  ' 

The  above  points  out  one  serious  potential  problem.  If  scores 
on  the  JPM  accurately  predict  success  in  ca^-**  out  the  actual  job 
tasks  in  a real-world  setting,  there  is  no  probelm.  This  job  perfor- 
mance  evaluation  is  an  excellent  opportunity  to  review  the  relationship  ' 

between  JPMs  and  actual  job  requirements.  From  this  review  can  cone 
recommendations  for  revision  of  any  JPMs  that  do  not  predict  successful 
job  performance. 

i 
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2,2.3  Collect  Questionnaire  Data 


Mailed  questionnaires  are  the  least 
expensive  procedure  for  collecting  data 
from  graduates  and  their  supervisors. 

However,  questionnaires  are  not  the  most 
reliable  method.  The  validity  of  mailed 
questionnaires  tends  to  reflect  how  well 
they  are  prepared  and  distributed  and 
what  portion  of  the  population  returned  them.  Also  the  degree  of  detail 
and  the  ability  to  follow  up  on  unclear  or  incomplete  responses  is  not 
as  great  as  with  the  personal  interview  approach. 

In  general,  more  specific  questions  are  asked  when  information  is 
gathered  with  a questionnaire.  This  is  partly  because  the  process  of 
consolidating  and  using  large  quantities  of  subjective  data  would  be  a 
formidable  task.  Also,  more  specific  questions  are  less  likely  to  be 
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misinterpreted  bii  those  who  are  asked  to  provide  the  information.  This 
is  iaportant  bi;cause  the  interviewer  will  not  be  there  to  clarify 
proolems. 

The  main  baJy  of  the  questionnaire  should  list  the  tasks  that  the 
graduate  was  trained  to  perforui.  The  supervisor  may  be  asked  to  note 
the  graduate's  ability  to  perform  these  tasks.  Also  the  graduate  might 
be  asked  to  note  his  ability  to  perform  the  tasks  and  to  rate  how  well 
the  instruction  equipped  him  to  perfonn  each  task.  Such  information 
as  frequency  of  task  performance  and  amount  of  supervision  required  might 
be  Included  in  the  questionnaire.  Open  ended  questions  could  be  included, 
asking  for  suggestions  on  how  to  improve  the  Instruction,  or  about  the 
equipment,  materials,  or  procedures  used  by  the  graduate. 

Valid  results  from  mailed  questionnaires  depend  largely  on  the  be- 
havior of  respondents.  A total  sanv)11ng  is  desirable.  Variations  In 
job  requirements  occur  because  of  command  requirements,  geographic  loca- 
tions, organizational  level,  etc.  Therefore,  make  certain  to  include 
all  graduates.  Since  not  all  students  and  supervisors  will  returri  the 
questionnaires,  a large  number  is  essential. 

2.2.4  Collect  Personal  Interview  Data 

Ideally,  personal  interviews  are  con- 
ducted by  specialists  from  the  quality 
control  team  who  are  familiar  with  the  par 
ticular  job  for  which  the  graduate  was 
trained.  They  visit  a representative 
sampling  of  graduates  and  obtain  first 
hand  data  on  graduate  assigrvnent. 
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utilization,  and  work  proficiency.  Data  Is  gathered  through  separate  in- 
terviews with  the  graduate  and  his  supervisor.  In  addition,  telephone 
Interviews  may  be  used. 

Make  a special  effort  to  clarify  to  both  the  graduate  and  his  super- 
visor why  the  Interviews  a>'e  being  conducted  and  what  will  Se  done  with 
the  collected  data.  If  they  realize  the  Important  part  they  play  in 
the  evaluation  process,  they  are  fTsore  likely  to  give  the  kind  of  Infor- 
mation needed. 

Section  2.1  of  this  block  lists  the  general  types  of  information 
needed  from  the  student.  Use  preplanned  lists  of  questions  and  try  to 
get  honest,  pertinent  answers.  The  Collect  Data  section  of  Block  1.1 
gives  suggestions  on  Interview  procedures. 

For  an  interviewer,  the  main  task  is  to  determine  graduate  pro- 
ficiency. But,  he  mi'st  also  be  concerned  with  how  the  graduate's  skills 
are  being  utilized,  and  how  well  the  graduate  is  progressing  through 
additional  training.  To  gather  these  data,  prepare  a list  of  supplement- 
ary questions.  Use  this  list  as  a guide  when  interviewing  the  graduate 
and  his  supervisor.  Of  course,  feel  free  to  alter  the  planned  sequence, 
rephrase  questions,  and  add  or  delete  questions.  Generally  fit  the  con- 
diict  of  the  interview  to  the  responses  obtained  from  the  person  being 
interviewed.  Accurately  record  answers  to  significant  questions. 

In  addition  to  the  above,  you  may  v;ish  to  administer  the  same  ques- 
tionnaires developed  for  Section  2.2.3,  assuming,  of  course,  that 
the  individuals  being  Interviewed  will  not  be  included  in  the  group 
who  will  receive  the  questionnaire  as  part  of  the  questionnaire  survey. 
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2.2.5  Obtain  Records  of  Students'  Performance  During  Instruction 
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The  internal  evaluation  resulted  in 
verification  of  cut-off  scores  for  the 
lesson  pos  tests  and  the  JPM.  In  the 
regular  operation  of  the  instruction, 
students  are  required  to  repeat  lessons 
if  they  fail  the  posttests  or  the  JPM. 

Records  of  students’  progress  through 

the  instruction  should  be  available  to  the  external  evaluator  in  case 
the  other  evaluation  data  indicate  that  students  are  not  pe. forming  well 
on  the  job. 

Figure  V.ll  shows  an  example  of  a data  sheet  that  gives  course  data 
for  each  studet  . participating  in  the  external  evaluation.  A pcsttest 
was  used  after  every  lesson  and  each  student’s  results  for  each  lesson 
is  shown.  At  the  top  of  the  lesson  column,  the  passing  criteria  is 
listed.  Some  tests  give  numerical  scores  and  a pass  criterion  while 
others  only  have  two  possible  scores--oass  or  fail.  Student  027  scored 
17  on  his  fourth  try  at  the  test  for  lesson  one,  module  A,  Unit  VII. 

He  needed  17  out  of  25  and  finally  oot  that  score.  Here  the  17  was 
his  score  and  the  tally  marks  show  the  number  of  times  plus  1 he  took 
the  test. 


The  primary  product  of  this  block  Is 


an  external  evaluation  report  (EXhK). 

This  report  Is  e summary  statement,  re- 
ferring all  pertinent  information  result- 
ing from  the  procedures  followed  In  this 
block,  of  the  external  evaluation  findings, 
their  Interpretation,  and  specific 
recommendations  for  revision  of  the  instruction  or  the  instructional 
system.  A summary  statement  of  the  previous  sections  of  this  block 
should  be  included  in  this  report.  The  procedures  for  consolidating 
data  and  making  revision  recommendations  follow. 

How  to  consolidate  data  into  a usable  format  has  been  covered  in 
Block  I.l.  Compile  separately  the  questionnaire  inputs  from  graduates, 
questionnaire  inputs  from  supervisors,  personal  interview  inputs  from 
graduates  and  from  supervisors,  and  JPMs.  Once  the  data  has  been  re- 
duced to  these  easier- to-handle  blocks  of  information,  begin  evaluating 
the  data  by  asking  some  pertinent  questions,  and  checking  the  data  for 
answers. 

Probably  the  first  question  to  ask  is: 

1.  Can  the  graduate  perform,  on  the  job,  the  tasks  he  was 
trained  to  perform,  at  the  planned  level  of  proficiency? 
Most  likely  there  will  be  some  conflicting  data.  For 
example,  most  graduates  may  say  they  can  perform  the  tasks 
while  most  supervisors  may  say  they  cannot. 
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This  leads  to  other  questions: 

2.  Precisely  which  tasks  are  not  being  performed  satisfact- 
orily? Look  for  the  answer  to  that  in  the  data  pool. 

3.  What  does  the  Job  performance  evaluation  data  say  about 
these  tasks  that  the  supervisors  claim  are  being  in- 
adequately performed?  If  the  JPMs  were  given  and  the 
evaluation  was  positive,  ki>ere  is  evidence  that  either 
the  JPM  or  the  work  procedures  in  this  command  are 
inadequate. 

Some  additional  questions  to  ask  are: 

4.  Are  the  tasks  that  the  supervisors  think  the  graduates 
should  be  able  to  perform,  the  same  tasks  that  were 
listed  in  the  original  job  analysis?  If  there  are  dis- 
crepancies, why  do  they  exist? 

5.  Are  there  tasks  for  which  the  supervisors  think  graduates 
were  inadequately  trained  that  were  on  the  original  task 
list  but  were  not  selected  for  training? 

6.  Are  there  areas  where  there  is  general  agreement  that 
while  graduates  can  score  well  on  JPMs,  they  cannot  do 
the  actual  task? 

The  above  should  illustrate  the  point  that  there  are  no  hard,  fast 
rules  for  evaluating  the  data  and  making  recommendations  for  change. 

If  most  graduates  and  most  supervisors  are  satisfied  with  the  quality 
of  training,  reconriend  that  few  if  any  program  changes  be  made.  At 
the  other  extreme,  if  a representative  sample  of  graduates  and  super- 
visors feel  something  went  wrong  between  the  initial  job  analysis  and 
the  final  training  program,  the  program  is  inadequate;  sift  through  the 
data  to  find  possible  causes  for  thie  inadequacy. 


If  many  graduates  cjnnot  perform  the  tasks  they  were  trained  to 
perform,  study  the  training  data  in  relation  to  the  other  data  gathered 
in  the  external  evaluation.  You  may,  for  example,  want  to  construct 
a table  of  the  kind  demonstrated  in  Block  V.l  to  see  whether  perfor- 
mance data  on  the  JPM  administered  at  the  end  of  the  course  can  be  used 
to  identify  graduates  who  cannot  perform  later  on.  This  would  be 
especially  important  if  the  two  JPMs  were  not  of  Identical  form.  You 
might  find  that  students  who  had  to  take  the  end  of  course  JPM  three 
or  more  times  before  passing  it  are  most  likely  to  fail  the  JPM  ad- 
ministered by  the  external  evaluators,  while  students  who  took  it 
only  once  or  twice  rarely  fail  .the  current  JPM.  In  such  an  instance 
you  mignt  need  to  re-examine  the  course  prerequisites  and  entry  be- 
haviors. 

Similar  tables  might  be  constructed  to  show  relationships  between 
supervisor  ratings  and  some  or  all  of  the  data  shown  in  Figure  V.ll. 

The  test  summary  data  at  the  bottom  of  Figure  V.ll  are  particularly 
useful  in  determining  required  changes  in  the  instruction.  At  the 
bottom  of  each  column  is  recorded  the  number  of  times  each  test  was 
given  and  the  number  of  students  who  passed  each  time.  Look  at  module 
A,  lesson  2 in  Unit  VII.  Notice  that  no  one  passed  the  first  time 
and  everyone  passed  the  second  time.  Results  like  that  are  suspect. 
Probably  there  is  either  a trick  to  passing  the  test  which  should  be 
included  in  the  lesson,  or  the  tester  is  not  scoring  the  same  each 
time. 

In  lesson  2 in  module  B,  Unit  VII,  everyone  passed  on  the  first 
trial.  Perhaps  the  test  was  a give-away,  too  easy,  or  there  was  too 
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(nuch  instruction  on  such  an  easy  unit.  The  opposite  Is  true  In  lesson 
2 of  module  A,  Unit  VIII,  no  one  passed  In  one  or  two  trials.  The 
students  may  pass  by  memorizing  the  test  rather  than  by  learning  what 
the  lesson  Is  designed  to  teach. 

How  bad  does  a program  have  to  be  to  cause  recomnendlng  revision? 
That  Is  a very  difficult  question  to  answer.  Fortunately,  In  the 
ISO  procedures,  If  you  can  pinpoint  the  problem  area  In  ISO  developed 
Instruction,  it  generally  can  be  fixed  without  disrupting  the  entire 
program.  Probably  the  best  single  guideline  Is  to  recomnend  program 
changes  In  areas  where  there  is  reasonably  high  graduate/supervisor 
agreement  that  a change  Is  needed,  provided  the  job  Is  done  according 
to  regulation.  Then,  In  areas  of  disagreement,  use  the  job  performance 
analysis  daU  to  determine  if  the  problem  is  Internal  or  external  to 
the  ISO  developed  course. 

As  a result  of  the  above  evaluation,  one  or  more  of  the  following 
types  of  recommendations  for  change  may  be  made: 

1.  Change  the  instructional  system. 

a.  Correct  task  selection  discrepancies  by  revalidating 
task  lists. 

b.  Correct  task  selection  errors  by  reevaluating 
selection  criteria. 

c.  Improve  JPM  validity. 

d.  Revise  instruction  to  fit  changes  made  in  the  analysis 
phase. 

2.  Change  the  job  structure. 

a.  Enforce  docui..entat1on;  that  is,  if  an  individual  is 
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assigned  to  a certain  job,  make  certain  the  job  he 
Is  actually  given  to  do,  matches  the  job  description, 
b.  Provide  necessary  support  such  as  tools,  equipment, 
etc.,  needed  to  perform  the  job. 

If  external  evaluation  shows  problems  with  the  instrucUonal  pro- 
gram, go  to  Block  V.3:  REVISE  SYSTEM,  and  make  the  necessary  changes. 
After  making  these  changes,  conduct  Internal  and  external  evaluation 
again.  Eventually,  major  errors  will  be  eliminated  and  changes  made 
will  be  only  normal  program  revision  based  on  changes  In  content  and 
doctrine. 

At  this  point  study  the  effectiveness  of  the  program  by  answering 
the  questions  asked  In  Section  2.1.2. 

1.  Do  a greater  percentage  of  graduates  of  the  current 
Instruction  show  satisfactory  performance  on  the  JPM 
than  graduates  of  older  courses  or  training  methods?  and 

2.  Do  graduates  of  the  current  course  require  less  on-the- 
job  training  than  graduates  of  other  courses  or  training 
methods? 

In  answering  the  first  question,  select  a group  of  subjects  from 
the  baseline  study  who  are  similar  to  the  graduates  of  the  instruction 
being  evaluated  in  education,  length  of  time  elapsed  since  training, 
military  experience,  etc.  Then  simply  compare  the  percentages  of  per- 
sons in  the  two  groups  who  perform  satisfactorily  on  the  .)PMs.  If  the 
level  of  satisfactory  performance  in  the  baseline  group  is  low  and 
the  graduates  being  evaluated  perform  well  (for  example,  40  percent 
success  in  the  baseline  group  and  85  percent  success  for  the  graduates 
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of  the  Instruction  being  evaluated),  you  probably  do  not  need  to  nake 
statistical  tests.  If,  however,  the  baseline  group  performed  well  so 
that  the  differences  were  small  (for  example,  75  verses  85  percent)  you 
will  want  to  make  a statistical  test  to  detennine  how  often  such  a 
difference  could  be  due  to  chance  alone.  You  should  obtain  h'-lp  from 
a research  and  evaluation  consultant  in  making  these  tests.  The 
answer  to  the  second  question  can  be  obtained  In  the  same  way  but  the 
data  will  come  from  the  questionnaires  and  1nte>“views  of  the  personnel 
and  supervisors  involved. 

Does  the  ISD  process  scop  here?  No,  it  never  stops.  The  fact  that 
today's  graduates  can  do  the  job  they  were  trained  to  do  does  not  mean 
tomorrow's  graduates  will  do  the  same.  The  job  may  change,  students 
may  change,  something  in  the  course  may  change;  the  qualities  of  instruc 
tors  may  change.  Assuring  optimum  training  quality  at  minimum  cost  de- 
mands a constant  feedback  of  information,  and  periodic  evaluation  of  the 
relationships  between  students,  the  instructional  program,  and  job  per- 
formance in  the  field. 

3.0  OUTPUTS 

The  outputs  of  this  block  should  consist  of: 

3.1  Products 

An  external  evaluation  report  (EXER).  This  report  is  a summary 


statement  of  the  external  evaluation  procedures,  findings,  interpreta- 
tion, and  revision  recommendations. 
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3.2  Other  Documentation 

Supporting  Information  Including. 

1.  A stat^nt  of  pertinent  Information  about  the  graduates 
Included  In  the  evaluation.  This  statanent  should  Include: 

a.  Nwber  of  graduates  evaluated 

b.  Graduates  pretraining  experience,  education  and  other 
pertinent  personal  Information 

c.  When  graduates  began  and  completed  the  particular 
course  being  evaluated 

d.  Statement  of  graduate's  work  activities  at  time  of 
evaluation 

e.  Statement  of  reasons  why  any  graduates  are  not  assigned 
to  the  duty  for  which  they  were  trained 

2.  When  external  evaluation  took  place.  How  much  time  had 
Intervened  between  completion  of  training  and  conducting  of 
external  evaluation 

3.  Statement  of  how  the  evaluation  was  conducted.  This  state- 
ment should  Include: 

a.  Number  and  qualifications  of  evaluators 

b.  Statement  of  how  data  was  collected 

c.  Pertinent  details  on  how  job  performance  evaluation 
was  conducted 

4.  A statement  of  graduate's  responses  to  mailed  question- 
naires and  personal  interviews 

A statement  of  supervisor's  responses  to  mailed  question- 
naires and  personal  Interviews 


5. 
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6.  A statement  of  results  of  Job  performance  evaluation 

7.  How  data  from  Items  4,  5,  and  6 above  were  Interpreted 

8.  Any  pertinent  information  not  listed  above,  that  In- 
fluenced final  recomwndatlons 

9.  Recommendations  for  change  in  Instructional  system 

10.  Recommendations  for  change  in  job  structure 

11.  Any  other  recommendations 


EXAMPLE 

Sanple  summary  of  External  Evaluation  Report  for  a course  in 
OH-58  Helicopter  Repair.  The  External  Evaluation  Report,  like 
the  Internal  Evaluation  Report  in  Block  V.l  is  too  extensive  for 
this  abbreviated  exan^le.  Many  conc1«<sions  can  be  drawn,  however 
from  the  coded  data  spread  sheet  in  Block  V.l  (page  36). 

Conci usions: 

1.  Thirty  percent  of  the  sample  of  students  had  previous  work 
experience  in  the  area.  These  students  performed  very  well 
on  the  entry  test,  within-course  tests,  and  performance  test. 
Tliey  also  tended  to  receive  very  hi^  ratings  from  both 
supervisors  and  peers. 

2.  Only  10%  of  the  student  san^ile  indicated  personal  interest  In 
the  area.  Student  performance  was  not  enhanced,  however,  and 
tended  to  be  on  the  lew  side  of  average. 

3.  Number  of  promotions  had  no  relationship  to  school  or  job 
perfortiance. 

4.  Students  who  performed  moderately  we^l  on  the  entry  test 
tended  to  perform  moderately  well  on  the  performance  test 
and  receive  high  ratings  from  both  supervisors  and  peers. 

5.  Saiety  records  reflected  no  relationship  to  performance 
in  school . 

6.  Supervisor  ratings  relate  very  strongly  with  peer  ratings. 

7.  All  trainees  were  assigned  to  the  appropriate  DOS. 
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FIGURE  V. 1C:  Flowchart  of  Block  Y.2;  CONDUCT  EXTERNAL  EVALUATION 
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OVERVIEW 


System  revision  mey  be  required  because  of  changes  in  doctrine  or 


content,  deficiencies  in  instruction  revealed  by  internal  evaluation 


or  external  evaluation,  or  requirements  for  naking  the  instruction  more 


efficient  in  terms  of  time,  money,  or  other  resources.  The  revision 


process  requires  empirical  data  for  determining  specific  areas  of  revision 
that  will  produce  optimum  pay-off  in  teme  of  increased  efficiency  or 


reduced  time  or  costs.  The  revision  may  consist  of  relatively  minor 


changes  to  specific  parts  of  the  system  or  may  involve  major  revisions 


of  the  instruction  and  procedures. 
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REVISE  SYSTEM 


1.0  INTRODUCTION 


The  final  function  In  the  ISO  process  is  that  of  maintaining  quality 
control  of  the  instructional  system  through  the  process  of  revision. 

Revision  here  means  any  systematically  planned  change  in  the  ISO  process 
based  on  operating  experience  and  data  such  as  that  ccllected  and  analyzed 
in  Blocks  V.l  and  V.2,  that  aie  subject  to  verification  through  continued 
evaluation.  Generally,  revision  means  those  changes  made  in  the  procedures, 
techniques,  and  materials  in  the  system.  T5ie  eirphasis  is  on  change  of 
technique  rather  than  of  content,  doctrine,  or  curriculum. 

In  this  block,  decisions  are  made  based  on  the  Inputs  from  Blocks  V.l 
and  V.2,  and  the  ccmparisions  of  these  reports  with  the  plans  of  the 
entire  system.  The  revision  process  is  characterized  by  analysis.  Inter- 
pretation, planning  and  follow-up.  The  inputs  are  analyzed  and  interpreted, 
those  elements  of  the  system  found  deficient  provide  the  basis  for  a revision 
plan  that  is  to  be  executed  according  to  an  established  schedule.  The 
follow-up  is  to  see  that  the  planned  changes  are  made  in  accordance  with 
the  plan. 

Since  a considerable  amount  of  work  and  effort  has  gone  into  the 
design  .ind  development  of  an  instructional  program,  it  is  fair  to  ask, 

"Why  revise?"  That  is  the  most  frequent  question  that  must  be  asked 
in  this  part  of  the  process.  Unless  it  is  necessary  previse.  It  is 
necessary  not  to  revise.  To  revise  for  the  sake  of  revision  is  not 
good  planning.  The  decision  about  what  to  revise,  if  at  all,  should  be 
made  after  careful  analysis  of  the  recommendations  submitted  in  the 
Block  V.l  and  Block  V.2  evaluation  reports. 
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As  with  most  other  features  In  a systems  model , the  reason  for 
revision  usually  Is  that  a clear  discrepancy  has  been  Identified.  It  Is 
ordinarily  the  size  of  this  discrepancy  that  causes  people  to  undertake 
revisions.  There  are  many  discrepancies  that  can  occur  In  any  of  three 
major  areas: 

1.  Needs 

2.  Internal  Results 

3.  External  Results 

Evaluation  could  reveal  a clear  discrepancy  between  Internal  needs 
and  the  Instructional  program  results.  Or,  there  could  be  a discrepancy 
between  the  external  needs  and  the  external  results.  Isolating  the 
problem  cannot  be  done  without  both  Internal  and  external  evaluation 
data.  Internal  evaluation  could  show  that  the  instruction  was  meeting 
tlie  planned  expectations;  the  trainees  could  do  what  they  were  taught 
to  do.  But,  the  results  could  be  judged  inadequate  by  external  evaluation: 
the  graduates  could  not  do  the  job  to  which  they  were  assigned.  It  is 
possible  that  some  instruction  will  result  in  overtraining,  but  discovering 
over- training  is  very  difficult. 

When  results  of  training  are  thought  to  be  inadequate,  the  evaluation 
process  must  be  designed  to  locate  and  evaluate  possible  causes.  Well- 
trained  personnel  may  be  sent  to  duty  assignments  and  through  impj'oper 
assignment  or  inadequate  supervision  not  perform  well  on  the  job.  It  may  be 
that  there  is  a delay  of  several  months  between  completion  of  training 
and  the  assignment  to  training-related  activities  in  the  duty  assignment. 

All  of  these  have  happened,  and  many  times  the  resulting  inadequate  job 
perfonnance  has  erroneously  been  thought  to  be  the  fault  of  the  training. 
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Although  there  are  no  absolute  rules  of  ='ssiqning  a particular  cause 
to  a particular  discrepancy,  the  relationship  of  the  training  to  the  actual 
job  assignment,  the  length  of  time  since  training,  the  availability  of 
supervision,  and  job  conditions  upf'n  which  training  assumptions  were  made 
should  certainly  be  taken  into  consideration.  As  with  most  other  changes 
in  tiie  instructional  program,  most  of  the  revision  requi recients  will  be 
generated  through  normal  changes  in  equipment,  tactics,  operational 
procedures,  restructuring  of  Defense  Occupation  Spewialties,  and  other 
specific  factors  relating  to  doctrine  and  operations.  Chances  brought 
about  by  normal  operations  can  be  more  easily  scheduled  and  can  become 
a p;,rt  of  normal  operating  procedure  for  the  various  departments  responsible 
for  the  instruction. 

There  are  three  major  sources  of  inputs  to  this  block: 

1.  The  Internal  Evaluation  Report  (INER)  from  Block  V.l 

^ 2.  The  Exterrial  Evaluation  Report  (EXER)  from  Block  V.2 

3.  Tile  System  Master  Plan  ei'her  from  Phas-;  .’II, 
or  as  specially  summarized,  from  Block  V.l. 

Internal  evaluation  reports  consist  of  specific  and  detailed  data, 
and  recommendations  for  revisions  based  on  evalu  itions  of  tiiese  data. 

The  INER  is  concerned  primarily  with  the  quality  and  achievements  of  the 
separate  components:  The  instruction,  the  tests,  the  degree  to  which 

the  instruction  adheres  to  the  Model,  the  learning  objectives,  and  a 
number  of  other  related  factors. 

The  EXER  deals  principally  with  the  relative  .success  of  the  graduates 
as  they  perform  on  the  job.  The  report  presumes  a careful  follow-up 
of  the  graduates  in  actual  work  assignments,  including  the  administration 


of  OPMs,  cuestionnaiires , Interviews,  and  responses  from  supervisors  and 
others  who  are  informed  about  the  graduate's  performance.  It  further 
so'’ves  as  a check  device  to  Insure  that  the  same  requirements  exist  in 
the  field  as  existed  when  the  Instruction  was  designed.  This  report 
which  is  an  output  of  Block  V.2,  contains  the  external  evaluative  findings 
aioiig  with  specific  identification  of  problems  and  recoimendatlons  for 
reyisions.  Tiiese  recommendations  are  based  on  an  analysis  of  field  user 
needs  rather  than  on  judgements  about  the  effectiveness  of  the  instruction. 

Ihe  system  master  plan  Includes  the  requirements  and  specifications; 
exactly  what  is  needed  and  expected  from  the  instruction  to  meet  the 
needs  of  the  using  comnands.  The  existence  of  this  control  document  is 
an  essential  requirement  of  the  revision  process.  It  is  the  base-line 
that  indicates  what  the  instruction  was  planned  to  achieve.  The  inpor- 
tanca  of  this  plan  carinot  be  overemphasized.  If  the  plan  is  not  documented 
and  not  a part  of  the  evaluation,  it  will  be  difficult  to  ever  establish 
factually  what  it  was  the  instruction  started  out  to  do.  This  action 
is  important  because  instruction  could  operate  for  years  and  the  original 

j designers  and  developers  be  gone  before  significant  decisions  are  made 

I 

based  on  externei  ev.iluacion  data.  It  is  important  to  know  what  was 
originally  planned,  rather  than  just  what  is  needed  now.  Having  such 
information  will  make  it  easier  to  understand  the  decisions  that  were 
made  early  in  the  program  development. 

The  process  of  making  revision  decisions  involves  the  analysis  of 
the  data  arfd  recommendations  from  tl.e  INER  and  EXER.  Following  this 
anat.vsis  of  facts  and  needs,  revisions  are  planned  in  terms  of  most 
imtia'-di ate  needs  consistent  with  available  resources. 
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There  will  never  be  enough  people  to  do  everything  that  could  be 
done  to  lo^rove  an  instructional  program.  Consequently,  a 11st  of  priori- 
ties must  be  developed  on  which  to  base  allocation  of  money  and 
personnel  resources  to  the  project.  Revision  may  mean  that  other  needed 
Instruction  will  not  be  developed  or  will  not  be  developed  on  the  original 
schedule.  Such  decisions  cannot  be  made  solely  on  the  basis  of  the 
performance  of  the  Instruction,  no  matter  how  good  or  how  bad  It  may  be. 

If  there  are  a number  of  interests  that  must  be  served  1n  making 
revision  plans,  perhaps  one  of  the  rating  or  ranking  methods  discussed 
in  Block  1.3  or  Block  II. 2 would  be  appropriate.  The  number  of  people 
whose  interest  must  be  respected  in  revisions  could  be  given  the  assign- 
ment of  rank  ordering  the  revision  requi renents . Perhaps  they  could  agree 
on  the  criteria  they  would  use  in  making  these  determinations.  This 
interdepartmental  decision-making  may  be  required  since  many  of  the  activities 
of  revision  will  cut  across  many  vested  interests. 

The  output  of  Block  V.3  is  a revision  requirements  report  which  is 
submitted  to  the  groups  that  will  acconplish  the  work  outlined  in  the 
report.  Representatives  of  these  groups,  working  as  a team,  murt  decide 
the  actual  work  schedule  and  resource  requirements  necessary  to  accomplish 
the  revisions.  Further,  a time  schedule  must  be  prepared  t*'  t shows  the 
needs  for  resources  through  the  entire  life  of  the  revision  process.  A de- 
tailed outline  of  how  to  prepare  the  revision  requirements  report  will 
be  presented  later  in  this  block.  The  steps  in  revising  the  system  are 
shown  in  the  flowchart  in  Figure  V.12,  the  fold-out  page  at  the  end  of 
this  block. 
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2.0  PROCEDURES 

2.1  Detennine  if  Revision  is  Required 
There  are  a nutnber  of  reasons  why 
revision  of  existing  Instruction  might  be 
considered.  The  major  reasons  are: 

1.  Change  In  doctrine  or  content. 

The  Defense  Occupation  Specialty 
(DOS)  rnty  be  changed  because  of 
changes  1n  canmand  organization, 
or  because  of  the  Introduction 
of  a new  weapon  system,  equipment,  etc 

2.  The  Internal  Evaluation  Report  (INER)  may  Indicate  that 
learning  objectives  are  not  being  met. 

3.  The  External  Evaluation  Report  (EXER)  may  Indicate  that 
graduates  are  not  able  to  do  the  job  for  which  they  supposedly 
were  trained. 

4.  Evaluation  of  the  instruction  or  changes  in  training  require- 
ments Indicate  a need  for  making  the  Instruction  more  efficient 
in  terms  of  time,  money,  or  other  resources. 

Fcr  some  of  the  items  listed  above,  the  decision  as  to  whecher  revision 


is  required  Is  relatively  simple;  for  others,  the  pros  and  cons  must  be 
carefully  balanced.  Following  is  a more  detailed  discussion  of  each  Item. 
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2.1.1  Making  Decisions  Based  on  Changes  In  Etoctrine  or  Content 

If  a DOS  Is  restructured  for  reasons  such  as  doctrinal  Changes,  or 
Introduction  of  new  procedures,  facilities,  weapons  systens,  etc.,  the 
related  instruction  clearly  will  have  to  be  revised.  If  the  tasks  required 
of  the  restructured  DOS  are  different,  the  revision  process  will  have  to 
begin  with  Block  I.l,  ANALYZE  JOB.  It  will  be  necessary  to  have  an  adequate 
analysis  of  the  revised  job  or  procedures  before  developing  instruction 
to  achieve  the  revised  objectives. 

When  jobs  change  or  DOSs  are  restructured,  tasks  selected  for  training 
In  any  given  DOS  are  directly  affected.  For  example,  suppose  instruction 
had  been  based  on  information  gathered  in  the  field  prior  to  the  consolida- 
tion or  restructuring  of  the  DOS.  In  addition  to  Block  I.l,  ANALYZE  JOB, 
another  likely  place  to  find  discrepancies  would  be  in  Block  1.2,  SELECT 
TASKS/ FUNCTIONS.  It  has  been  Indicated  repeatedly  that  those  tasks  rated 
as  high  priority  for  on*'  DOS  may  be  only  "nice  to  know"  In  a related  DOS. 
After  the  DOS  restructuring,  the  priorities  may  change  considerably.  In  this 
case,  it  would  be  necessary  to  go  through  the  selection  process  again, 
making  sure  that  appropriate  representatives  for  all  of  the  affected 
DOSs  were  included  in  the  selected  sample.  Adding  and  deleting  tasks  in 
the  restructured  DOS  may  completely  re-order  the  priority  of  all  the  tasks 
on  the  list.  The  task  list  should  be  considered  as  a total  entity  and 
individual  tasks  should  not  be  added  or  deleted  without  careful  revision 
of  all  tasks. 

Since  this  process  is  a technical  one,  it  should  only  be  undertaken  by 
those  who  are  familiar  with  the  process  and  preferably  by  those  who  did 
the  original  work  on  the  DOS. 
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2.1.2  Making  ttecislons  Based  on  the  Internal  Evaluation  Report 


If  the  Internal  Evaluation  Report  (INER)  points  out  that  students 
are  not  meeting  the  learning  objectives,  sone  degree  of  revision  will 
be  essential.  Either  the  tests  are  too  stringent,  or  they  are  testing 
tlie  wrong  things,  or  the  instruction  Is  Inadequate  for  the  parti cula>" 
students  entering  the  course.  If  the  students  who  have  been  entering 
the  course  are  representative  of  those  who  must  be  trained,  either  the 
tests,  the  instruction,  or  both  must  be  revised. 

It  should  be  en^jhasl2ed  at  the  beginning  that  revising  Instruction  does 
not  mean  the  same  thing  as  adding  Instruction.  Inadequate  performance  In 
some  areas  of  the  course  will  require  careful  analysis  to  find  the  cause 
of  the  problem,  and  revision  of  the  Inadequate  portions  imist  be  based 
on  conclusions  about  the  causes  of  the  pniblems.  While  revising  Instruc- 
tion may  Involve  adding  Instruction,  this  Is  not  always  the  case. 

2.1.3  Making  Decisions  Based  on  the  External  Evaluation  Report 


I£  the  External  Zvalu'tlon  Report  (EXER)  shows  that  graduates  who 
are  promptly  assigned  to  the  duties  for  which  they  were  trained  are  not 
able  to  perform  these  duties,  the  course  imist  be  revised.  However,  note 
that  the  above  statement  contained  a big  "IF."  Some  implications  of 
this  are: 


1.  If  the  graduate  was  not  assigned  tc  the  duty  for  which  he 
was  trained  until  six  months  or  more  after  completion  of 
the  Instruction,  the  problem  Is  probably  with  the 
assignment  rather  than  with  the  Instructional  program. 
Perhaps  training  can  be  postponed  until  the  Individual  Is 
in  line  for  imnediate  assignment  to  the  duty. 
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2.  If  the  graduate  was  trained  for  one  Job  but  assigned  to  a 
different  one,  the  Instructional  program  cannot  be  binned 
for  his  Inactequate  performance. 

3.  If  the  graduate's  supervisor  requires  the  graduate  to  perform 
tasks  that  are  not  officially  considered  a part  of  the  Job 
to  which  the  graduate  Is  assigned,  something  Is  wrong  somewhere 
In  the  conmand  system.  But  it  would  be  erroneous  to  assume 
that  the  instructional  program  was  inadequate. 

Only  careful  analysis  of  the  EX£R  data  will  make  possible  a realistic 
decision  as  to  whether  the  proper  tasks  have  been  selected  for  training, 
and  whether  individ'.'als  who  meet  the  learning  objectives  can  reasonably 
be  expected  to  perform  the  job  tasks. 

2.1.4  Making  Decisions  Based  on  Efficiency  of  Instruction 

Probably  an^  Instruction  can  be  made  more  efficient;  that  Is,  if  you 
are  willing  and  able  to  pay  the  price.  The  real  consideration  here  is 
whether  the  pay-back  from  increased  instructional  efficiency  will  be 
greater  than  the  cost  of  making  it  more  efficient. 

There  is  a funcintental  assumption  in  ISD  in  this  area  that  has,  over 
a wide  number  of  cases,  tended  to  hold  up.  It  is  not,  however,  specific 
to  any  given  instruction  and  there  are  no  specific  laws  or  equations 
that  can  be  written.  However,  simply  stated,  instruction  should  be  revised 
when  the  time  or  cost  to  revise  (investment)  will  be  repaid  by  the  Improved 
time  cost  performance. 

Suppose  that  in  Block  III. 5,  it  was  decided  that  the  instruction 
was  sufficiently  valid  to  be  used.  And  suppose  the  decision  was  based 

3 

s 

S; 


f i 


97 


In  part  on  the  fact  that  there  were  alternative  or  repetitive  Instructional 
pathways  to  provide  remedial  or  repeat  Instruction  1n  those  areas  where 
students  did  not  reach  criterion  on  the  first  exposure.  Once  this  decision 
has  stood  for  a while  (through  a number  of  students)  Its  appropriateness 
again  can  be  brought  up  for  consideration. 

If  the  nunt>er  of  students  who  repeat  segments  of  the  Instruction 
Is  fairly  high,  and  the  required  amount  of  Instructor  time  Is  high,  revising 
the  materials  in  order  to  Increase  the  efficiency  may  be  profitable.  This 
assumes,  of  course,  that  the  Instruction  was  effective  and  that  the  students 
were  meeting  the  criteria  after  second  or  third  testings.  If  the  students 
were  net  meeting  the  criteria,  then  It  is  obvious  that  revision  efforts 
must  be  undertaken  anyway. 

The  principle: 

INSTRUCTION  CAN  BE  MADE  MORE  TIME  EFFICIENT  THROUGH 
EMPIRICALLY  BASED  SYSTEMATIC  REVISION, 
is  probably  sound  for  a11  instruction  that  has  not  been  developed  and 
revised  by  ISO  processes.  However,  each  time  it  is  Improved,  it  moves 
closer  to  its  extreme  limit  of  itr^i-ovement.  For  example,  1f  instruction 
is  improved  by  a factor  of  SOS  of  Its  original  time  during  the  first 
major  revision,  this  same  percentage  improvement  cannot  continue  In 
successive  revisions.  There  are  both  procedural  and  arithmetic  reasons 
for  this  limit. 

The  arithmetic  reasons  are  simple.  If  instruction  takes  100  hours 
and  is  reduced  by  50%,  it  now  takes  50  hours.  If  It  is  reduced  by  60% 
again,  it  would  take  0 hours  if  the  original  time  Is  used  as  the  basis 
for  comparison.  If,  on  the  other  hand,  the  new  base  of  50  hours  is 
used,  and  that  1s  reduced  by  50%,  the  reduction  is  actually  25  hours. 
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That  ?5  hours  is  only  25%  of  the  original  base.  So,  on  successive  revisions, 
only  smaller  and  smaller  real  reductions  can  be  obtained.  The  first  iter- 
ation reduction  was  50  hours,  the  second  reduction  was  25  hours,  and  If  It 
were  done  again  at  the  same  rate,  the  reduction  would  be  only  12.5  hours, 

The  technical  reasons,  while  not  so  obvious  and  systematic,  are, 
nonetheless.  Important,  The  state-of-the-art  Is  simply  not  able  to 
produce  the  same  percentage  of  Improvement  on  successive  revisions.  There 
Is  a tendency  to  use  up  all  of  the  good  procedures  for  reducing  time 
early  In  the  revision  cycle,  and  much  more  costly  and  difficult  procedures 
are  required  to  make  continued  Improvements.  Some  of  these  procedures 
would  have  to  be  tested  In  the  actual  environment  and  might  or  might 
not  work. 

Propeller  aircraft  in  the  late  1930 's  and  1940 's  became  faster  and 
faster  and  could  carry  heavier  payloads  for  longer  ranges.  Gradually, 
they  reached  their  limit.  If  it  Is  assumed  that  450  mph  was  a realistic 
maximum  speed,  when  the  top  speed  of  existing  aircraft  was  only  250  mph, 
there  was  a strong  possibility  of  making  gains  with  new  models,  However, 
eventually  these  gains  were  made.  It  Is  difficult  to  imagine  much  of 
an  improvement  over  an  F-51  or  an  AD,  other  than  special  purpose  modifi- 
cations. They  could  be  made  safer,  more  reliable,  gradually  Increasing 
range,  payload,  or  speed  through  trade-offs,  but  they  were  about  as  good 
a.«i  they  were  going  to  get  without  a major  breakthrough.  The  F-86  and 
the  F9F  were  dramatic  changes  due  to  fundamental  improvements  In  power 
plant  technology.  They  were  not  "improved,"  they  were  drastically  changed. 
They  represented  a major  breakthrough. 


IMfortunately,  sudi  u major  breakthrough  In  educational  techM>1ogy 
cannot  presently  be  offered.  However,  It  is  not  necessarily  true  that 
instruction  that  has  been  nade  more  efficient  or  less  costly  cannot  be  made 
more  effective.  Or,  if  it  is  affective,  it  can  probably  be  made  more 
efficient  or  less  costly,  unless  it  has  been  revised  several  times.  These 
factors  must  be  taken  into  account  during  original  design  as  well  as 
during  the  revision  process.  Think  of  the  trade-offs  necessary  to  arrive 
at  a correct  mission  configuration  for  an  aircraft;  range,  payload,  and 
speed.  The  same  principle  of  trade-offs  applies  to  design  and  revision 
of  instruction. 

Figure  V.13  is  most  tn^rtant  for  the  revision  decision.  It  shows 
that  for  an  equally  effective  instruction,  the  mmber  of  revisions  it 
goes  through  determines  the  percentage  of  time  reduction  that  can  be 
attained.  If  the  original  version  was  a traditional  course,  and  the 
first  revision  was  ISO,  the  time  savings  probably  would  be  greater  than 
that  shown  In  the  figure.  The  major  point  here  1s  that  if  time  savings 
are  important,  they  are  most  likely  to  occur  after  only  a few  revisions: 
after  that  there  is  little  excess  Instruction  left  to  reduce.  If  a course 
has  reached  somewhere  between  505  and  70S!  of  original  time,  there  Is 
little  chance  of  further  time  reductions  after  the  fourth  or  fifth  iteration 
and  revision. 

The  key  here  is  the  nuntier  of  times  that  adequate  ISO  revisions  have 
been  made,  not  the  number  of  students  who  have  gone  through  the  1nst^uct^on 
or  the  number  of  classes  graduated.  This  point  cannot  be  over-emphasized, 
Each  time  substantial  gains  are  made  on  the  basis  of  revision,  the  chances 
for  making  as  imich  of  a gain  on  the  next  revision  are  less. 
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FIGURIi:  V.13:  T1n«  Reductions  in  ISO  Courjcs  as  a Function 
of  the  Nuinber  of  Revisions.  (This  assumes  ISO 
is  applied  to  an  existing  course  which  is  100% 
time.) 

One  should  conclude  that  revisions  made  early  ought  to  be  as  thorough 
and  complete  as  possible  so  as  to  make  as  much  of  the  total  possible 
gain  as  can  be  made.  Saving  ?0%  of  100  hours  Is  much  trore  desirable 
than  saving  20%  of  50  hours,  since  it  Is  usually  the  actual  number  of 
hours  saved  that  makes  the  difference,  not  the  percentage  of  change  from 
the  base. 

The  tin\e  and  effort  needed  to  reduce  the  titm-  in  the  course  are  added 
costs.  They  should  have  the  characteristics  of  inve~,tr;.ent,  and  that 
investment  will  either  prove  to  have  been  a good  use  of  resources  or  not 
a good  use  of  resources,  depending  on  the  results.  Usually  a good  invest- 
ment in  course  efficiency  requires  that  a large  ntxnber  of  stuck;nts  go 
through  the  course. 

Cost  per  student  is  another  dimension.  Traditional  forms  of  instruc- 
tion which  use  a relatively  constant  teacher-student  ratio  will  result  in 
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an  almost  linear  cost  based  on  the  number  of  stu(tents,  Courses  which  have 
b^n  done  according  to  principles  of  ISO,  in  which  cost  was  a planning 
constraint,  should  produce  economies  similar  to  those  shown  graphically 
if!  Figures  V.14  and  V.15. 
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FIGURE  V.14:  Average  Total  Cost  in  ISO  and  Traditional 
Instruction  as  a Function  of  the  Number  of 
Students . 
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MltfffiER  OF  STUDENTS 

FIGURE  V.15:  Life  Cycle  Costs  for  Traditional  and  ISO 
as  a Function  of  the  Nuober  of  Students 


Investments  In  educational  technology  mAy  lead  to  reductions  In  the 
average  cost  per  graduate  In  a training  program.  These  investments  can 
be  justified  If  the  cost  reductions  stentning  from  the  adoption  of  new 
instructional  methods  exceed  their  added  costs.  The  tools  of  Instruc- 
tional technology  are  usually  so  characteristic  of  the  more  capital 
Intensive  technologies  that  they  usually  cannot  be  acquired  and  Implemented 
on  any  sort  of  continuum.  They  must  be  acquired  In  discrete  units  and 
often  with  capacities  which  far  exceed  those  needed  for  training  systems 
with  small  throughput.  Furthermore,  they  are  usually  highly  specialized 
and  technically  efficient.  Scale  economies,  which  are  characterized  by 
a reduction  In  the  average  costs  for  graduates,  are  often  very  significant 
for  the  larger  training  systems. 

The  analysis  of  average  student  costs  Is  Illustrated  In  Figure  V.14. 

In  this  Instance,  the  average  total  costs  are  equal  for  systems  at  N2. 

For  systems  which  have  more  students  than  N2  (e.g.,  N3),  the  ISO  system 
Is  less  expensive,  while  at  Nj,  It  is  more  expensive. 
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The  previous  discussion  is  a static  analysis  and  does  not  represent 
3 life  cycle  cost  estimate.  A life  cycle  cost  estimate  imjst  consider 
the  incidence  of  costs  and  time,  and  discount  each  increment  of  cost  to 
determine  that  system  with  the  lowest  present  cost  which  represents  the 
most  efficient  alternative.  Figure  V.15  demonstrates  this  relatlonsiilp. 

Th';  c cal  costs  of  a traditional  system  and  an  ISO  system  are  illus- 
trated by  the  functional  relationship  of  and  TCj^q.  The 

initial  investment  for  the  traditional  system  (OA)  is  less  than  for  the 
ISO  system.  However,  the  ISO  system  has  lower  variable  costs  and  for  a 
system  with  a capacity  for  more  than  N2  students,  the  ISD  system  becomes 
less  expensive, 

2.2  Determine  What  Needs  Revision 

Certainly  ycu  should  not  begin  revising 
until  the  cause  of  the  problem  has  been 
carefully  identified  and  verified.  The 
person  in  charge  of  making  revision  decisions 
must  be  sure  that  the  causes  and  recommendations 
in  the  INER  and  EXER  are  well-founded.  It  's 
this  independent  review  o<"  the  findings  that 
gives  this  block  its  uniqUe  importance.  That 
is  why  the  functions  of  instruction,  evaluation,  and  making  revision 
decisions  have  been  stated  and  placed  separately  in  the  model.  They 
all  provide  a system  of  checks  and  balances  and  the  opportunity  to  be 
independent  of  each  other  in  the  eyes  of  management.  As  these  Indepen- 
dent functions  are  separately  evaluated,  each  group  involved  should  be 
held  responsible  only  for  its  function.  Instructors  should  not  be  held 
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accountable  for  bad  design,  evaluators  should  not  be  responsible  for 
bad  results,  and  revisers  should  be  completely  Independent  of  it  all 
to  permit  as  much  objectivity  as  possible. 

2,2.1  General  Guidelines  for  Cteciding  What  to  Revise 

The  question  of  “what  needs  revision?”  Is  a critical  one.  To  answer 
this  question,  a number  of  factors  must  be  considered  In  the  context  of 
the  total  operation  of  the  course.  Certain  blocks  in  the  Modet  can  be 
identified  as  being  high-,  medium-,  and  low-payoff  processes.  Blocks  I.l 
and  1,2  are  examples  of  potentially  lilgh-payoff  steps  In  terms  of  dollars, 
time,  and  validity.  Any  revisions  made  in  these  blocks  impact  the  entire 
course.  Changes  In  the  outputs  of  Blocks  I.l  and  1.2  will  require  the 
addition  or  deletion  of  segments  of  the  course. 

On  the  other  hand,  changes  in  the  media  probably  will  result  in  minimum 
changes  in  validity  and  time  and  will  be  reflected  mostly  in  cost  changes. 
Thus,  changes  made  early  in  the  ISO  process  ordinarily  result  in  greater 
differences  in  cost,  time,  and  effort  than  changes  made  later  in  the 
process . 

This  should  not  be  interpreted  to  mean  that  you  should  begin  at  the 
end  of  the  process  and  work  toward  the  frontl  Recently,  during  the 
process  of  developing  instructional  materials,  some  trainees  were  no', 
doing  well  on  the  posttest.  The  designers,  exercising  one  of  their  options, 
simply  revised  the  test;  greatly  Improving  the  “quality  of  the  instruction." 
It  was  cheap,  quick,  and  had  the  desired  outcome.  But  the  logic  was 
the  same  as  adding  a quart  of  water  to  a quart  of  milk  and  saying  there 
were  now  two  quarts  of  milk.  There  were  two  quarts  of  something,  but 
it  would  be  illegal  to  call  it  milk. 
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The  most  direct  source  of  information  that  will  be  useful  in  deciding 
vjhat  to  revise  will  be  the  INER  and  EXER.  As  the  ins<:n<ctiona1  designer 
gains  mere  experience,  the  decisions  n.v.de  based  on  this  data  should  get 
better  and  bettor.  However,  now,  some  aspects  of  the  course  are  much 
more  readily  and  accurately  evaluated  than  others.  Mention  was  made 
earlier  that  internal  evaluation  was  more  accurate  than  external  evaluation 
because  the  test  instruments  are  more  accurate.  Performance  tests,  time, 
job  task  data,  anti  student  achievement  are  all  more:  directly  evaluated 
than  are  external  factors  such  as  job  performance  under  a wide  range 
of  field  conditions.  If  students  are  having  problems  with  a certain  part 
of  the  couric,  the  problem  usually  Is  more  easily  found  and  corrected 
than  if  supervisors  are  not  satisfied  with  what  students  are  learning 
in  the  coui*£e.  This  is  true  because  student  performance  measures  are 
more  direct  than  are  supervisor  ratings.  In  supervisor  ratings,  much 
of  the  error  can  be  in  the  supervisor. 

2.2.2  Considerations  in  Revising  Operating  Instructions 

As  in  many  processes,  many  of  the  problems  in  deciding  what  to  revise 
boil  down  tc  practical  everyday  issues:  Time,  people,  and  money.  Once 
instruction  is  operational,  It  usually  has  to  stay  operational  since  the 
needs  for  trained  people  continue  to  exist.  This  means  the  instruction 
must  be  revised  during  the  time  it  is  still  being  offered.  And,  since 
everyone  is  usually  fully  occupied  with  its  operation,  there  are  few, 
if  any,  pe.  "'le  to  do  revision  work.  On  the  bright  side,  if  everything 
has  been  done  in  the  original  design  to  make  the  instruction  as  modular 
as  possible,  revision  becomes  easier  since  the  problems  can  be  isolated 
and  the  instruction  revised  one  piece  at  a time.  Further,  if  self-paced 
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instructiofi  Is  used,  students  can  often  be  rescheduled  to  give  some  relief 
and  time  for  revisions. 

It  Is  also  possible  to  substitute  1nstruLl:ors  for  modules  Mh11e  the 
module  Is  being  revised.  The  opposite  Is  rarely  true.  Instructors  may 
resist  being  “replaced"  by  modular  instn  ctlon  or  self-instruction,  also 
It  takes  time  to  develop  the  modules.  If  the  instruction  uses  self-pacing 
and  peer  tutoring,  for  example,  the  early  finishers  can  be  taught  the 
revised  infomation  or  procedures  and  then  can  be  used  to  teach  the  new 
material  to  others  while  the  modules  are  being  revised. 

2.2.3  Considerations  in  Attempting  to  Reduce  Time 

The  alternatives  already  discussed  often  work  well  when  the  problem 
is  found  to  be  a change  in  doctrine  or  content.  Other  problems  may  require 
more  creative  solutions.  Suppose  the  instruction  were  self-paced  and 
modularized,  but  the  problem  was  that  it  was  taking  too  much  time.  Suppose, 
for  e.xample,  the  trainees  were  not  getting  to  their  duty  stations  soon 
enough.  Shortening  instruction  can  be  accomplished  by  removing  some 
of  the  content,  but  instruction  developed  by  the  ISO  process  should  not 
have  content  that  can  be  eliminated  unless  field  conditions  have  been 
changed.  Also,  one  of  the  design  features  of  the  ISO  Mode!  is  to  eliminate, 
where  possible,  subjective  decisions  and  to  emphasize  data-based  objective 
decisions.  For  example,  assume  it  is  hignly  desirable  to  reduce  the 
length  while  keeping  the  original  level  of  performance  (effectiveness). 

Tp  do  this,  we  must  apply  the  concepts  of  Value  Engineering  to  the  ISO 


process. 


A basic  assumption  is  that  noting  Is  "froe."  Evor>'  accompli shntnt 
requires  time,  money,  or  other  resources.  So,  to  "value  engineer"  ISO 
Instruction,  tine  and  effort  must  be  spent  In  finding  ways  to  achieve 
the  same  results  In  less  tine,  and,  hopefully,  without  Increasing  the 
total  costs.  !t  Is  not  always  possible  to  Increase  efficiency  without 
Increasing  costs,  but  those  decisions  must  be  made  deliberately. 

In  or<ter  to  revise  Instruction  to  reduce  student  time,  It  Is  necessary 
to  have  good  data  on  tiie  length  of  time  required  by  the  trainees  to  complete 
eadt  current  segment  or  module.  There  are  rules  of  thumb  for  good  places 
to  look,  ftie  possibility  Is  that  there  are  bottlanccks  in  the  system; 
that  Is.  places  where  students  nmst  spend  considerable  time  waiting  their 
turn  to  use  materials  or  equipment,  or  waiting  to  be  tested  or  processed 
by  the  Instructors  In  some  way. 

The  time  spent  In  waiting.  In  the  bottleneck,  can  possibly  be  eliminated 
by  rearranging  the  conditions,  using  more  equipment,  adding  more  instructors, 
using  peer  tutors,  or  reesigning  the  procedure.  Or,  perhaps  the  bottleneck 
cannot  be  eliminated.  Then,  It  might  be  possible  to  use  the  tinie  wasted 
In  the  bottleneck  for  other  productive  purposes.  Perhaps  the  physical 
space  can  be  used  for  Individuals  to  begin  orientation  tp  subsequent 
Instruction,  or  perhaps  they  can  be  assigned  other  required  duties.  A specific 
solution  is  hard  to  predict,  but  the  process  Involves  locating  the  problem 
and  then  generating  a number  of  possible  solutions  until  one  Is  found 
that  most  closely  solves  the  problem.  Sometimes  It  Is  useful  to  seek 
opinions  from  others  who  are  unfamiliar  with  the  specific  situation  and 
who  can  ask  “naive"  questions  such  as;  “Why  do  you  have  them  do  that?" 

•'Why  do  they  do  this  before  that?"  Being  requ1>'ed  to  answer  these  questions 
titty  turTi  up  aitematives  not  previously  considered. 
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Since  there  are  rarely  obvious  and  easy  solutions  to  such  problems, 

It  may  be  necessary  to  redesign  the  segn^ts  of  instruction  that  currently 
take  the  most  time.  Since  people  tend  to  chink  In  complete  units:  full 
days,  weeks,  months;  It  Is  often  possible  to  find  partf  of  days,  weeks 
or  even  months  which  are  not  fully  used.  If  It  normally  takes  four  and 
a half  days  to  do  a part  of  the  Instruction,  It  1s  usually  difficult 
to  begin  a new  unit  on  Friday  afternoon,  rinding  ways  to  save  thesf; 
formerly  unuse  1 segments  of  time  can  i.ake  a worthwhf'e  reduction  In  the 
time  required  to  complete  the  instruction. 

If  none  of  these  procedures  produces  the  desired  results.  It  may 
be  necessary  to  redesign  larger  segments  of  instruction.  This  could 
require  chances  in  the  media,  the  manageinent  system,  the  testing  procedures, 
and  other  factors.  Development  of  different  testing  procedures  is  possible 
If  there  Is  sufficient  need  to  do  so,  but  such  development  takes  time 
and  requires  collecting  a great  deal  of  data.  Past  history  has  Indicated, 
however,  that  the  possible  tirite  javings  are  quite  high. 

Virtually  all  of  these  methods  apply  most  directly  to  instruction 
that  is  modularized  and  self-paced.  Block  scheduled,  televised,  or 
platform-instructed  courses  prese? t another  kind  of  problem.  It  is 
normally  not  possible  to  have  the  instructor  talk  faster  or  to  play  the 
television  tapes  at  a high  rate  of  speed.  While  much  instruction  is 
presented  in  a block-scheduled  manner,  it  unfortunately  is  the  most 
difficult  kind  of  instruction  to  shorten  without  eliminating  subjecc 
matter.  That  is,  it  is  the  most  difficult  to  shorten  if  the  management 
plan  and  delivery  system  are  to  remain  the  same.  If  the  method  of  deliver- 
ing the  instruction  can  be  changed,  all  of  the  potential  savings  inherent 
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In  the  alternative  delivery  systea  may  be  realized.  Going  from  group- 
based  Instruction  to  Individual -based  Instruction  offers  a potential 
saving  In  the  time  required  because,  as  has  been  mentioned  earlier,  some 
students  learn  faster  than  others  and  It  Is  the  tine  of  the  faster  learner 
that  often  can  be  saved  and  returned  to  the  system.  If  everyone  Is  scheduled 
In  block-fashion,  the  faster  learners  still  learn  faster,  but  there  Is 
no  way  to  recover  this  time  for  use  elsewhere. 

2.2.4  Considerations  In  Attempting  to  Improve  Instructional  Effectiveness 

One  of  the  findings  In  the  INER  may  be  that  the  Instruction  Is  not 
effective;  it  does  not  meet  the  planned  objectives.  If  the  EXER  Indicates 
a moderately  inadequate  level  of  training  of  graduates,  there  Is  a need 
to  locate  specifics  In  the  EXER  and.  If  possible,  track  these  through 
to  the  INER.  If  graduates  cannot  perform  tasks  3,  5,  and  0 on  the  Job, 
and  If  there  has  been  difficulty  teaching  these  tasks  at  the  school, 
you  need  to  begin  analysis  with  these  three  tasks.  Troubleshoot  the 
course  through  the  use  of  ISD  logic. 

To  begin,  assiwne  that  the  EXER  is  correct:  The  graduates  really 
cannot  perform. 

Unless  the  graduates  failed  the  test,  it  can  be  assumed  It  Is  at 
least  part  of  the  problem.  No  matter  how  bad  the  Instruction,  If  the 
posttest  works  properly.  It  will  not  permit  students  to  graduate  If  they 
are  not  trained. 

One  of  the  following  could  be  wrong  with  the  posttest. 

1.  The  test  lacks  validity.  It  does  not  separate  those 
students  who  can  do  the  work  from  those  who  cannot. 
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2.  The  test  Is  so  specific  that  students  can  learn  how  to 
pass  the  test  becai»e  they  learn  from  re-taking  the 
test.  They  ran  pass  the  test  on  specifics,  but  can^'iot 
generalize  the  knowled^. 

Ihitll  this  problem  with  the  posttest  Is  resolved,  there  Is  no  point 
In  doing  additional  work  on  the  Instruction.  Specifically,  careful  study 
of  the  outputs  of  Blocks  1.3,  II. 2.  and  the  testing  procedures  In  IV. 2 
should  be  examined.  There  Is  a ^od  chance  that  reviewing  these  testing 
procedures  will  lead  to  a solution  of  the  problem  since  these  data 
Indicate  how  the  test  worked  under  actual  field  conditions. 

Cnee  the  test  has  been  revised.  It  should  be  used  Instead  of  the 
first  version,  with  enough  students  to  find  out  whether  It  Is  working. 

Next,  reexamine  the  EXER.  Were  all  graduates  Inadequate,  or  Just 
some  fraction  of  them?  If  there  were  some  satisfactory  graduates  and 
some  unsatisfactory  ones,  and  If  the  posttest  had  a numerical  score, 
find  out  whether  those  who  scored  highest  on  the  posttest  were  found 
to  be  satisfactory  graduates.  If  so,  the  battle  Is  half  won.  If  not, 
continue  to  use  the  revised  posttest  until  there  are  enough  graduates 
In  the  field  to  do  another  follow-up. 

If  there  is  not  a validated  JPM  to  use  as  a msans  of  measuring  Job 
performance,  the  problem  1s  more  difficult.  Without  the  JPM,  there  are 
only  supervisors'  ratings  and  other  soft  data  that  normally  are  not 
highly  reliable.  That  is,  two  qualified  supervisors  will  not  give  the 
graduate  the  same  performance  rating.  If  supervisors  cannot  agree  on 
who  the  satisfactory  graduates  are.  It  is  very  hard  to  t;'a1n  Ltudonts 
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to  meet  these  varied  expectations.  Much  faith  will  have  to  be  placed 
In  the  external  evaluator's  Judgement.  He  will  Interview,  collect  data. 
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and  make  recommendations  for  revisions  based  on  the  data  and  on  his 
Impressions.  But,  no  one  will  be  able  to  say  for  sure  whether  the  changes 
will  result  in  a more  acceptable  graduate.  !t  will  be  a trial  and  error 
process. 

Many  factors  will  determine  the  course  of  action  taken  if  there  are 
no  validated  JFMs.  If  the  school  environment  has  a high  degree  of 
fidelity  with  the  job  world,  the  development  of  JPMs  may  be  the  necessary 
first  step.  Follow  the  procedures  In  Block  1.3.  Remember  that  high 
fidelity  Instruction  In  school  is  more  likely  on  those  tasks  In  technical, 
clerical,  administrative,  and  certain  operator  skills  where  the  actual 
equipment  Is  available  at  the  school. 

For  certain  DOSs,  high  fidelity  with  the  job  will  be  difficult  to 
achieve,  e.g.,  Hospital  Corpsman,  Infantryman,  and  a revision  approach  may 
have  to  be  based  on  an  agreement  between  the  schools  and  the  using 
commands.  If  representatives  of  the  schools  and  using  commands  agree 
on  a task  list  and  level  of  proficiency  for  graduates,  the  revision  can 
start  at  that  point.  These  meetings  and  discussions  may  be  the  best 
way  to  resolve  differences,  particularly  where  the  notion  that  the  school 
should  produce  a totally  trained  person  can  be  discussed. 

Further,  discussions  based  on  data  and  interviews  could  lead  to 
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discoveries  that  tasks  have  been  changed  or  that  supervisors  did  not 
understand  the  objectives  of  the  course.  In  these  cases,  a much  better 
common  ground  of  agreement  can  be  established. 
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The  EXER  could  also  have  shM«/n  that  student  graduates  believed  they 
were  overtrained  for  the  Job  while  supervisors  believed  they  were  under- 
trained, If  students  were  trained  to  do  tasks  they  were  never  allowed 
or  asked  to  do,  and  not  trained  cn  some  tasks  expected  of  them,  such  a 
discrepancy  could  exist,  This  discrepancy  might  have  to  be  tolerated 
since  en  entire  course  cannot  be  revised  based  on  that  kind  of  data. 
However,  If  virtually  all  using  commariJs  and  trainees  are  consistently 
In  agreement  on  those  questions  and  on  which  tasks  were  not  required, 
revision  would  be  In  order.  It  would  also  mean  an  Inc^ctequate  task  analysis 
had  been  conducted. 

2.3  Prepare  Revision  Plan 

The  preparation  of  a revision  plan 
Involves  the  analysis  of  the  evaluation  data 
on  a block-by-block  and  step-by-step  basis, 

A revision  plan  format  Is  presented  In 
Figure  V,16,  The  purpose  of  the  revision 
plan  Is  to  identify  the  problems  in  each 
step  of  the  process  and  to  Indicate  the 
intended  action  to  be  taken  to  eliminate 
the  discrepancies.  The  plan  should  Include 

clear  statements  of  the  problem  to  be  solved  and  the  time  and  resources 
required  to  meet  the  goals. 

In  order  to  gain  inputs  and  alternatives  from  those  most  familiar 
witfi  the  processes,  the  revision  plan  U then  discussed  with  each  of  the 
people  whose  work  will  be  affected  by  It.  It  may  be  that  a revision  plan 


REVISION  PLAN  FORMAT 


1.  Courso  identification  Informtlon. 

Tralninr;  requirements  being  met  by  the  course.  Number  of  students 
being  trained,  over  what  periods  of  time,  to  what,  degree  of  proficiency, 
in  what  DOS. 

3.  Present  costs  of  meeting  training  requirements.  Instructional  person- 
nel requirements , facilities,  equipment,  time,  etc.  Estimated  costs 
of  my  cn-the-job  training  due  to  inadeouacy  of  present  program. 

4.  Why  course  revision  is  recommended. 

a.  Based  on  INER? 

b.  Based  on  E.XER? 

c.  Based  on  changes  to  job  structure? 

d.  Based  on  potential  savings  of  time  or  money? 

5.  Specific  details  of  deficiency  in  present  course. 

6.  Specifically  what  changes  are  recommended. 

a.  What  is  rationale  for  specific  recommendations? 

b.  What  are  costs  of  revision? 

c.  Time  schedule  for  revisions. 

d.  Intact  of  revisions  on  presently  operating  course. 

e.  Personnel  requirements  for  revision. 

7.  Estimated  costs  of  meeting  training  requirements  with  revised  course. 

8.  Alternative  plans  considered  and  reasons  for  rejection. 

9.  Projected  funjre  requirements  for  course. 

10.  Probable  results  if  course  is  not  revised. 

FIGURE  V.16:  Revision  Plan  Format 
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cannot  be  adequately  prepai*ed  without  prior  consultation  with  the 
specialists  in  the  various  steps  In  the  process. 

One  Important  point  here  Is  that  enpliasis  should  be  placed  on  reaching 
agreement  between  the  schools  and  the  using  cormands  based  on  sound  job 
analyses  and  a clear  statement  of  conniand  needs.  The  fewer  the  assumptions 
made  about  what  the  using  commands  need  and  the  more  direct  the  communication, 
the  more  likely  will  be  acceptance  of  the  graduates  when  they  arrive 
after  training. 

The  revision  plan  should  contain  a good  estimate  of  the  Implications 
that  making  the  revisions  will  have  on  other  on-going  work.  Course 
revisions  must  fit  Into  the  work  priority  schedule  just  as  any  other  work 
must  fit.  The  more  carefully  the  plan  is  made,  the  better  the  decisions 
about  priorities  can  be  made. 

The  kinds  of  improvements  that  are  planned  should  be  clearly  stated. 
Decreasing  time  by  applying  what  specific  methods?  Increasing  the  pro- 
portion of  students  passing  the  tests  on  the  first  atteirpt  may  be  possible, 
but  there  must  be  reason  to  believe  that  the  proposed  improvements  will 
actually  make  a difference  that  outweighs  the  costs.  The  benefits  of 
the  Improvements  can  be  estimated,  as  well  as  the  penalties  for  not 
making  the  improvements.  This  information  can  be  used  to  make  decisions 
about  where  to  allocate  training  resources  to  do  the  most  good  for  the 
total  training  system,  not  for  just  one  unit  of  Instruction. 


2.4  Perform  Follow-Up  Activity 

Once  the  determination  is  made  to  revise  — = — 

ttie  system,  personnel  must  be  assigned  ^ I j 

responsibility  for  the  various  revision  steps.  t 

Generally,  the  individuals  responsible  for  ' — | 

each  of  the  steps  in  the  ISO  process  will  >— t — 

undertake  the  required  revision  effort.  ■■■ 

If  the  revision  plan  requires  only  a (3 

simple  change  in  adnini strati on  or  a test 
or  a part  of  the  instruction,  revision  may 
be  as  simple  as  using  the  revised  part 

again  and  evaluating  the  results.  However,  if  the  first  item  in  the 
revision  plan  is  to  conduct  job  analysis,  and  if  this  new  Job  analysis 
results  in  a task  list  that  is  different  from  the  original  task  list, 
all  the  blocks  in  the  ISO  Model  will  have  to  be  followed.  However, 
fortunately,  if  the  ISO  process  was  followed  when  the  instruction  being 
revised  was  originally  developed,  irajch  valuable  information  will  be  avail- 
able to  assist  in  the  revision  effort. 

In  Block  V.l,  CONDUCT  INTEWJAL  EVALUATION,  progress  evaluation  and 
p;  Kess  evaluation  were  discussed.  The  information  given  there  is  a 
good  set  of  guidelines  for  following-up  or  the  system  revision  activity. 


3.0  OUTPUTS 


The  outputs  of  this  block  should  consist  of  the  following: 
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3. 1 Products 

A conplete  revision  plan  following  the  general  fonnat  given  In 
Figure  V.16  (page  113)  of  this  block. 

3.2  Other  Documentation 

r Follow-up  Information  including  the  following: 

r 1.  Revision  progress  vs.  revision  plan 

2.  Reasons  for  any  deviation  from  the  plan 

3.  Specific  results  of  each  major  step  In  the  revision  plan 

I 
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EXAHPLE 


Sample  Revision  Plan  for  a Course  In  OH-58  Helicopter 

tepalr-- 

1.  Course  Identification  Information: 

A.  Trains  students  to  perform  user  support  level  main- 
tenance of  the  OH-58  helicopter  airfrane,  power  plant, 
hydraulic  systems.  Instruments,  electrical  system, 
turbine  engine  maintenance  and  change,  rotor  systems, 
power  train  system,  flight  controls.  Inspections, 
and  avionics. 

2.  Training  requirements: 

A.  The  course  produces  90  OH-58  Helicopter  Repairmen 
each  8 weeks,  all  qualified  to  perform  the  above 
support  level  user  maintenance  undr-r  supervision. 

This  excludes  inspection  and  avionics,  each  of  which 
must  be  supplemented  with  two  weeks  of  FOOT. 

3.  Present  costs  of  meeting  training  requirements: 

A.  The  course  required  8 instructors,  8 support  personnel, 
administrative  office  space  for  8,  instructors'  offices 
for  8,  2 classrooms  seating  100,  and  one  maintenance 
area  containing  8 OK-58  helicopters. 


B. 


4.  Reason  for  course  revision  recommendation: 


A.  Recommendations  based  on  INER. 

1)  Revise  units  and  #5;  Within-course  posttest 
scores  below  standard, 

2)  

B.  Recommendations  based  on  EXER, 


1)  Revise  safety  instruction:  Accident  rate  has 
increased  9%. 


2) 


K 
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5.  Specific  details  of  deficiency  In  present  course: 

A.  Consistently  W student  performance  on  unit  #2 
Indicated  a need  for  revision.  Analysis  of  the 
Instruction  in  unit  #2  Indicates  that  the  topics 
imist  be  bt'oken  down  Into  smaller  units  of  skill 
and  Information. 


6.  Specific  changes  recommended: 

A.  Unit  #2: 

1)  Ur, it  on  Installation  of  main  transmission  thermo- 
switch  win  be  subdivided  into  tiiree  separate 
sub-uni  ts : 

a)  Inspect  main  transmission  themK>sw1tch 

b)  Install  main  transmission  thernoswl tdi 

c)  Procure  main  transmission  thermoswltch 


7.  Estimated  costs  of  meeting  training  requirements: 
A.  Equipment  costs: 

1}  IOC  general  mechanics  tool  kits  @ $35.00 
per  k1t--$3,600 
2)  


8.  Alternate  plans  considered  and  rsasois  for  rejection: 

A.  The  proposal  to  move  training  on  the  airframe, 
flight  controls,  and  instruments  to  FOOT  was 
rejected.  These  systems  are  interdependent  with 
several  other  major  systems  and  cennot  be  ignored 
during  formal  school  training.  Only  inspection 
and  avionics  represent  independent  topics.  They 
will  be  trained  through  FOOT. 

9.  Projected  future  requirenents  of  the  course: 

A.  The  department  of  the  Army  Headqurrters  does  not 
anticipate  an  increased  need  for  OH’  53  Helicopter 
Repairmen  during  the  next  24  months. 


1 


B. 


ISM'tiKJt:’ 


REFEBFNCES 


PHASE  V 


Block  V.l 

Department  of  the  Air  Force.  Handbook  for  designers  of  instructional 
systems:  Evaluatlo.i  (Vol.  V,  AFP  Washington  D.C.: 

Headquarters » United  States  Air  Force,  July  1974. 

This  fifth  volume  in  the  Air  Force's  series  of  pamphlets  on 

instructional  design  is  devoted  to  evaluation  of  training  oro- 

grams.  To  provide  for  evaluation  of  the  instructional  svstem, 

the  last  step  in  the  Air  Force  Model,  an  evaluation  plan  is 

nreoared.  Under  this  plan,  the  evaluation  plan  is  conducted  bv 

personnel  who  are  not  involved  in  planning  or  conducting  the 

training.  This  plan  specifies  both  internal  and  field  evaluation. 

Kaufman,  R.  A..  Educational  system  planning.  Englewood  Cliffs,  N.J.: 
Prentice-Hall , Inc.,  1972. 

One  of  the  elements  of  the  systems  aporoach  to  instructional 
design  that  distinguishes  it  from  other  aporoaches  is  the  eval- 
uation-revision cycle.  The  use  of  this  cycle  allows  the  de- 
signer to  develop  a "self-correcting"  mechanism  whereby  instructional 
products  are  revised  until  they  meet  specific  performance  criteria 
as  identified  in  the  objective. 


Preceding  page  Wank 
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Swezevt  R.  W.,  aiKi  Pearlsteln,  R.  B.  Developing  criverlon-referenced 
tests.  Reston,  Va,:  Applied  Science  Associates,  19/4,' 

The  development  of  pretests,  posttests  and  the  testing  olan  Is 
a requisite  component  in  the  data  collection  stage  of  process 
evaluation.  To  be  effective  Indicators  of  the  outcomes  of 
Instruction  It  Is  necessary  to  construct  criterion-referred 
tests  which  measure  a trainee's  ability  to  perform  a specific 
task  at  a specific  level  of  performance. 

Thorndike,  E.  L.  (Ed.).  Educational  measurement.  Washington: 

American  Council  on  Education,  T9?l. 

Educational  evaluation  seeks  to  produce  data  of  various  sorts 
which  can  be  used  for  the  purpose  of  educational  decision-making 
and  ultimately  the  improvement  and  refinement  of  specific  in- 
structional products.  One  helpful  article  in  this  collection 
is  "The  Evaluation  of  Educational  Programs"  by  A.  M.  Astin  and 
R.  J.  Panes.  In  this  article,  the  Inputs,  outputs,  and  operations 
of  an  evaluative  study  are  reviewed  In  regard  to  their  relative 

j importance  to  the  decision  making  process. 

|1 

i 

; Tracy,  W.  R.  Designing  training  and  developirient  systems.  New  York: 

American  Manag^ent  Association,  Inc. 

The  purpose  of  conducting  an  internal  evaluation  is  to  collect  data 
and  information  in  order  to  Improve  the  training  system.  If  the 
evaluation  is  to  be  valid,  al’  components  of  an  instructional  system 
must  be  analyzed;  Instructors,  trainees,  instructional  materials, 
strategies,  facilities  etc. 
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PHASE  V 


Block  V.2 

Kaufman,  R.  A.  Educational  system  planning.  Englewood  Cliffs,  N.J.: 
Prent1ce-Hall , Inc.,  1972. 

The  systems  approach  requires  members  of  the  instructional  design 
team  to  become  proficient  In  the  tools  of  assessment  and  evalua- 
tion. A systems  approach  to  instructional  design  is  noted  for 
its  constant  evaluation  and  revision  until  instruction  is  effec- 
tive in  realizing  its  goal  of  efficiently  trained  men  being  success- 
ful on  the  job. 

Swezey,  R.  W.,  & Pearlstein,  R.  B.  Developing  criterion-referenced 
tests.  Reston,  Va.:  Applied  Science  Assoc iatos , 

A criterion-referenced  test  measures  an  individual's  performance 
compared  to  some  external  criteria  or  performance  standard.  As 
such,  it  can  be  used  to  evaluate  the  effectiveness  of  an  instruc- 
tional program  designed  to  train  students  on  specific  tasks  which 
are  critical  to  the  trainee's  success  when  he  enters  the  field. 

Thorndike,  E.  L.  (Ed.).  Educational  measurement.  Washington,  D.C.: 
American  Council  on  Education,  Id!?!. 

Thorndike's  collection  of  articles  on  measurement  offers  several 
articles  on  evaluation.  One  useful  article  is  "Performance  and 
Product  Evaluation"  by  Robert  Fitzpatrick  and  Edward  Morrison. 
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In  performance  and  product  evaluatlon.some  criterion  .ituatlon 
such  as  vocational  (job  performance)  capabilities  are  assesr-ed. 

A “work  sample”  test  evaluates  some  component  of  a trainee's 
on-the-job  performance  after  training  has  been  completed. 

Tracy,  W.  R.  Designing  training  and  development  systems.  New  York: 
American  Management  AssocVatTjn,  Inc. 

The  ultimate  test  of  any  instructional  system  is  the  trainees' 
ability  to  perform  successfully  on  the  job.  The  external  evalu- 
ation or  follow-up  of  a training  program  is  concerned  with  an 
evaluacion  of  the  products  of  the  training  program.  The  external 
evaluation  collects  data  pertaining  to  the  quality  of  the  job 
perfcrmances  of  the  trainees.  Tracey  devotes  a chapter  to  evaluating 
training  systems. 

Tracey,  U.  R.,  Flynn,  E.  B.,  & Legere,  C.  L.  J.  The  develotOTent  of 
instructional  systems  procedures  manual.  Fort  Devens,  Mass.. 

United  States  Aimy  Security  Agency,  1970. 

Conclusive  proof  of  the  adequacy  of  an  instructional  system  can  be 
obtained  only  by  follow-up  and  evaluation  of  trainees  on  the  job. 
Methods  of  evaluation  are  discussed  in  several  chapters  of  this 
document,  along  with  suggestions  for  conducting  the  evaluation. 
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Kaufman,  R,  A.  Educational  systan  planning^  Englewood  Cliffs,  N.J.: 
Prentice-Hall,  Inc.,  197Z. 

The  results  of  evaluation  may  yield  data  Indicating  that  the 
course  output  Is  not  meeting  the  Intended  course  goals.  The  re- 
vision process  or  'juallty  control  refers  to  any  systematically 
planned  change  of  the  Instructional  system.  To  ensure  system 
responf Iveness  It  Is  necessary  to  carry  out  a planned  revision 
when  such  a need  Is  apparent. 

Short,  J.  G.,  Geear,  L.  G.,  Haughey,  B.  E.,  S Tien,  D.  T.  Strategies 
of  training  development:  Final  report  of  a pro.iect  to  develop  a 
fundamentals  course  (AIR-E-9?-^/68-FRyT  Pittsburgh,  Pa.:  American 
Institutes  for  Research,  February  1968, 

A course  was  designed  to  teach  electronics  troubleshooting  skills. 
The  course  was  developed  in  a series  of  gradual  approximations  of 
the  final  version  of  the  cou»*se.  Through  successive  revisions 
based  on  trainee  on-the-job  performance  a uniformly  high  level  of 
performance  was  reached  by  all  trainees.  The  results  of  tiris 
project  provide  good  evidence  of  what  careful,  systematic  revision 
can  accomplish. 
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ACRONYMS 

AFS  - Air  Force  Speciality 

AR  - A^t^y  Regulations 

CMI  - Computer  Managed  Instruction 

CRT  - Criterion  Referenced  Test 

CODAP  - Comprehensive  Occupational  Data  Analysis  Programs 

DOS  - Defense  Occupational  Specialities 

FN  - Field  Manuals 

FOOT  - Formal  On  The  Job  Training 

GEO  - General  Educational  Development 

HQ  - Headquarters 

ISO  - Instructional  Systems  Development 
ISS  - Installation  Support  School 
ITV  - Instructional  Television 
JPA  - Job  Performance  Aids 
JPM  - Job  Performance  Measure 
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KOR  - Knowledge  of  Results 
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LO  > Learning  Objective 
LS  - Learning  Step 

mOB  - Military  Occupational  Data  Bank 

HOS  - Military  Occupational  Specialities  (Army/Harine  Corps) 

MiO  “ Modification  Work  Orders 
NIH  - Not  Invented  Here 

NOTAI'  ~ Naval  Occupational  Task  Analysis  Program 

OJT  - On  The  Job  Training 

OSR  - Occupational  Survey  Report 

POI  - Program  of  Instruction 

QQPRI  - Qualitative  and  Quantitative  Personnel  Requirenents  Identification 

RS  - Resident  School 

SME  - Subject  Matter  Expert 

SMP  - System  Master  Plan 

SOP  - Standing  Operation  Procedures,  Standard  Operating  Procedures 
STEP  - Self-Teaching  Exportable  Package 
TAK  - Trainer  Appraisal  Kit 
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TI  - Traditional  Instruction 
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TLO  - Terminal  Learning  Objective 

TM  - Technical  Manuals 

TOE  - Tables  of  Organization  and  Equlfvnent 

TRADOC  - U.S.  Training  and  Doctrine  Comnarid 


- ^ ^ liii.jiiuiiiSiailiffllliiiiililliliLj 


GLOSSARY 


ABILITY  GROUPING;  Arrangement  whereby  students  are  assigned  to  groups 
on  the  basis  of  aptitude  testing. 


ABSOLUTE  STANDARDS:  A statanent  defining  the  exact  level  of  performance 
required  of  a student  as  a deimnstration  that  he  has  mas- 
tered the  course  objective{s).  Criterion-referenced  tests 
are  usually  based  on  an  absolute  standard. 


ACHIEVEMENT  GROUPING:  Arrangement  whereby  students  are  assigned  to 

groups  according  to  their  performance  on  pretests  of  units 
of  the  course. 


ACTION:  Occurs  in  terminal  learning  objectives  and  learning  objectives; 

describes  the  specific  behavior  the  learner  is  to  exhibit 
after  training. 


ACTION  VERBS:  Verbs  that  convey  action  and  reflect  the  type  of  learn- 
ing that  Is  to  occur.  Action  verbs  must  reflect  behaviors 
that  are  measureable,  observable,  verifiable^  and  reliable. 


ACTIVITY  STEP;  One  simple  operation  or  movement  that  comprises  part  of 
a job.  A job  performance  standard  consists  of  a list  of 
these  operations  or  movements. 


ADJUNCT  PROGRAMMING:  A method  of  combining  the  features  of  good  exist- 
ing instructional  materials  (e.g.,  films,  textbooks)  with 
special  directions  or  questions  to  guide  the  learner. 


ADMINISTRATIVE  CRITERIA:  In  media  selection,  the  options  that  course- 

ware be  developed  locally  or  at  some  central  location. 


ALGORITHM;  A rule  or  procedure  for  accomplishing  a task  or  solving 
a problem. 


ALPHANUMERIC:  Refers  to  a combination  of  letters  and  numbers;  for 

example,  on  the  keyboard  of  a teletype. 


ALTERNATE  PATH;  Refers  to  elements  which  have  relationships  in  which 
the  specific  situation  encountered  determines  the  appro- 
priate sequence,  or  it  may  be  another  way  of  meeting  the 
same  objective. 
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ASSESSMENT:  A judgment  of  the  effectiveness  and  efficiency  of  a 

training  system,  in  terms  of  measurement  and  evaluation. 


ASSOCIATION  DEVICES:  Menwry  aids,  techniques  which  ease  recall. 
Mnanonic  devices. 


ATTITUDE;  A persisting  state  of  a person  that  influences  his  choice 
of  action. 


ATTITUDE  MEASURE:  An  instrument  designed  to  gather  information  about 
how’ people  feel  toward  a paTicular  object.  This  could 
include  liking  or  disliking  subject  matter,  usefulness 
of  a medium,  or  opinions  about  the  medium. 


AUDIO-ONLY  PROGRAM:  A production  which  does  not  contain  any  video 
or  pictures;  for  example,  a record  or  radio  program. 


AUDIO  PRODUCER:  Prepares  tape  recordings  and  produces  audio  programs. 
The  audio  producer  combines  narration,  music,  and 
other  sound  effects  in  the  production  of  an  audio  pro- 
gram. 


AUDIOVISUAL  MEDIA:  Refers  to  any  device  such  as  television  or  film 

which  is  both  seen  and  heard. 


BASELINE  DATA:  Valid  and  reliable  information  about  the  current 

level  of  performance  of  the  intended  student  population. 
This  data  can  be  used  to  confirm  the  need  to  develop  new 
instruction,  or  can  be  used  as  a comparison  in  ascer- 
taining differences  between  students'  performance  be- 
fore and  after  instruction. 


BEHAVIORAL  ATTRIBUTES:  Qualities  or  activities  that  cha*-'-'  -:erize  an 

object  or  process.  Behavioral  attributes  characterize 
each  category  of  learning. 


BLOCK  SCHEDULING:  Mode  of  instruction  whereby  all  students  receive 
the  same  instruction  at  the  same  time. 


BLOCKING:  Refers  to  tne  process  of  defining  and  illustrating  the 

different  camera  movements  and  camera  shots  in  a tele- 
vision or  film  script.  A blocked  script  may  also 
contain  directions  as  to  the  movement  of  actors  as  v/ell 
as  scenery  changes. 


CHECKLIST:  Job  performance  aid  which  lists  the  elements  of  a task 
inthe  sequence  of  execution.  The  job  holder  places 
a check  beside  each  element  as  it  is  accomplished,  thus 
insuring  that  the  task  is  completed. 


CHRONOLOGICAL  ORDER;  Arranging  concent  in  order  from  one  topic  to 
another  based  on  when  they  occurred  in  time. 


COMMON-FACTOR  LEARNING  OBJECTIVES;  Refers  to  learning  objectives  that 
are  identical,  or  that  have  identical  action  words  and 
similar  objects  of  the  action  in  the  naming  objective 
statement. 


COMPARATIVE  SEQUENCE:  Sequencing  which  starts  with  familiar  topics 
and  goes  to  unfamiliar  ones. 


COMPLEXITY  CRITERION;  In  media  selection,  the  degree  of  complexity 
required  of  instructional  materials  in  order  to  ade- 
quately train  students  to  meet  learning  objectives. 


COMPUTER  MODELS  TECHNIQUE:  Occurs  during  the  simulation  of  an  opera- 

tional system;  involves  having  a computer  simulate  the 
major  operations  of  the  system,  under  a variety  of  condi 
tions. 


CONDITIONS:  Occurs  in  terminal  learning  objectives;  describes  wfiat 
is  presented  to  the  student  in  order  to  accomplish  the 
specified  action,  that  is,  it  describes  the  important 
aspects  of  the  performance  environment. 

CONTIGUITY:  Refers,  in  1 earning , to  the  principle  that  events  which 

occur  closely  together  become  associated  by  the  learner. 


CONTINGENCY  MANAGEMENT;  The  establishment  of  a set  of  procedures  by 

which  trainees  are  required  to  perform  a certain  amount  of 
work  or  to  achieve  certain  objectives  before  engaging  in 
activities  that  are  preferred  by  the  trainee  (e.g.,  recrea- 
tion, a break,  or  a more  desirable  training  event). 
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COURSE  DOCUMENTATION:  Information  describing  the  current  concert 

of  a course  (instructional  materials,  tests  instructor's 
manual,  evaluation  plan,  student's  manual)  and  its  develop- 
mental history  (job  analysis,  criteria  for  selecting  tasks 
for  training,  previous  revisions). 


CLUSTERING:  A process  of  organizing  many  tacks  into  groups  for  the 

purpose  of  deciding  upon  the  optimal  instructional  set- 
ting mix  for  that  group  of  tasks. 


CRITERION-REFERENCED  TEST:  Measures  what  an  individual  can  do  or 
knows,  compared  to  what  ne  must  be  able  to  do  or  must 
know  in  order  to  successfully  pe'^form  a task.  Here  an 
individual's  performance  is  compared  to  external  cri- 
teria or  performance  standards  which  are  derived  from  an 
analysis  of  what  is  required  to  do  a particular  task. 


CRITICAL  CUE:  Cue  which  must  be  correctly  interpreted  by  the  student  be- 

fore we  can  correctly  perform  the  associated  task. 


CKiiiCAL  SEQUENCE:  Sequencing  of  topics  or  objectives  according  to 
their  importance. 


CUE:  A word  or  other  signal  that  initiates  or  guides  behavior;  a 
prompt. 


CUT-OFF  SCORE:  Minimum  passing  score. 


DATA:  Collection  of  facts  or  numerical  values  resulting  from  observa- 

tions of  situations,  objects,  or  people. 


DATA  COLLECTION  PLAN;  An  outline  of  the  procedures  and  techniques 
that  will  be  ujed  to  gather  infomation  for  any  specific 
purpose. 


DATA  RECORDING  PLAN:  Method  of  tabulating  background  responses  and 

test  data. 


DECAY  RATEij-  The  amount  of  time  it  takes  a trainee  to  forget  what  iie 
j has  learned  in  school.  If  the  decay  rate  is  high  then  a 
trainee  should  not  receive  instruction  in  a specific 
task^ntil  shortly  before  he  will  actually  perform  it. 
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DECISION  TREE:  Flowchart;  graphic  representation  of  the  sequence  of  a 

specific  activity  or  operation. 


DELIVERY  SYSTEM:  Any  method  containing  plans  and  procedures  for  the 
presentation  of  instruction.  Platform  instruction,  tele- 
vision, FOOT,  and  STEPs  are  all  delivery  systems. 


DEPENDENT  RELATIONSHIP:  Occurs  when  skills  and  knowledges  in  one 
'learning  objective  are  closely  related  to  those  in  the 
other  learning  objective.  In  order  to  master  one  of  the 
learning  objectives,  it  is  first  necessary  to  learn  the 
other. 


DOWNTIME:  Refers  to  the  period  of  time  when  equipment  is  inoperable. 


DUTY:  One  of  the  major  subdivisions  of  work  performed  by  one  indi 

vidual.  One  or  more  duties  constitute  a job. 


DUTY  TITLE:  Categorizes  groups  of  tasks  under  identifiable  head 
ings  to  help  in  the  organizing  of  lists  of  tasks. 


EMPIRICALLY  BA:1ED  REVISION:  Revision  based  on  the  results  of  test 
da\a  and  the  collection  of  other  types  of  quantitative 
information. 


ENTRY  BEHAVIOR:  The  skill,  knowledge,  and/or  attitude  required  be- 

fore beginning  a new  segment  of  instruction;  also  may 
refer  to  the  capability  a person  has  prior  to  new 
learning. 


ENTRY  SKILLS:  Specific,  measurable  behaviors  that  have  been  determined 
through  the  process  of  analysis  of  learning  require- 
ments to  be  basic  to  subsequent  knowledge  or  skill  in  the 
course. 


ENTRY  SKILLS  TEST:  A measurement  instrument  designed  to  determine  if 

a student  already  possesses  certain  skills  or  knowledge 
needed  as  a prerequisite  before  undertaking  new  instruc- 
tion. 


ENTRY  TEST;  Contains  items  based  on  the  objectives  that  the  intended 
students  must  have  mastered  in  order  to  begin  the  course. 
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tRROR  OF  HALO:  Occurs  when  an  observer  sometimes  allows  his  rating 
of  performance  to  be  influenced  by  his  general  impres- 
sion of  a person. 


ERRORS  OF  LOGIC:  Occur  when  two  or  more  traits  are  being  rated.  It 
is  present  if  an  observer  tends  to  give  similar  ratings 
to  traits  which  do  not  necessarily  go  together.  The 
traits  are  related  only  in  the  mind  of  the  person  making 
the  error. 


ERRORS  OF  STANDARD:  Occur  when  observers  tend  to  rate  performers  too 
high  or  too  low  because  of  differences  in  their  standards. 


EVALUATION:  The  process  of  interpreting  the  results  of  measurement 
data  (e.g.,  tests,  JPMs)  for  the  purpose  of  making  a 
judgment  or  decision  on  the  Instruction  or  on  the  success 
of  a trainee. 


EVALUATION  CRITERIA:  The  measures  used  to  determine  the  adequacy 
of  performance. 


EVALUATION  PLAN:  A method  or  outline  of  what  set  of  procedures  will 
be  used  to  gather  data  and  information  for  the  purpose 
of  assessing  a course  of  instruction. 


EXTERNAL  CUES:  Signals  for  action  that  exist  outside  of  the  student 

(conditions,  features,  or  characteristics  of  the  job 
environment  that  trigger  action). 


FALSE  NEGATIVE:  Occurs  when  a person  can  perform  t!ie  task  but 

receives  a failing  score  on  the  test. 


FALSE  POSITIVE:  Occurs  when  a person  cannot  perform  the  task  but 

receives  a passing  score  on  the  test. 


FEEDBACK:  The  return  of  information.  Information  on  student  per- 

formance is  "fed"  back  to  the  student  so  that  he  car* 
improve  that  performance;  to  the  instructional  designer 
so  that  he  can  improve  materials  and  procedures  on  the 
basis  of  student  needs;  to  the  management  system  sc  it 
can  monitor  the  internal  and  external  integrity  of  the 
instruction  and  make  appropriate  revisions.  Or,  refers 
to  the  flov/  of  data  or  information  from  one  step  in 
the  ISD  Model  to  others. 
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FOOT--FORMAL  ON-THE-JOB  TRAINING:  This  type  of  training  takes  place 
in  the  actual  work  situation. 


FOLLOW-UP  ACTIVITIES:  The  work  events  that  occur  after  a course  of 
Instruction  has  been  completed. 


FORMATIVE  EVALUATION:  The  iterative  process  of  developing  and 
improving  instructional  materials  and  procedures. 


FIDELITY:  Refers  to  how  well  the  actions,  conditions,  cues,  and 
standards  of  the  JPM  approximate  those  of  the  task. 


FIELD  USER  NEEDS:  The  general  and  specific  duties  that  will  have 
to  be  taught  to  the  trainee  if  he  is  to  be  able  to 
adequately  perforin  in  a real  world  environment. 


FIRST  DRAFT  MA.TERIAI.S:  Any  materials  (book,  film,  etc.)  which  are 
not  yet  committed  to  their  final  form.  First  draft 
refers  to  the  fact  that  the  materials  are  still  in 
'rough'  form  and  will  be  revised  on  the  basis  of  test 
results  and  other  data. 


FLOWCHART:  A graphic  representation  of  the  sequence  of  a specific 
activity  or  operation;  decision  tree. 


FRONT  END  ANALYSIS:  Refers  to  job  analysis,  selection  of  tasks  for 

training,  and  development  of  JPMs. 


FIXED  SEQUENCE:  Refers  to  elements  that  are  always  done  in  the  same 
order. 


GRAPHIC  ARTIST;  Designs  and  prepares  a wide  variety  of  visual  illu- 
strations such  as  graphs,  charts,  and  diagrams. 


GRAPHIC  S Measurement  device  which  includes  some  type  of  number 

line  on  which  students  indicate  their  attitude  toward 
a social  object. 


GO  NO-GO:  Pass-fail;  criterion  of  evaluation  whereby  student  can- 

not be  "partially  correct".  He  is  either  100%  correct 
(go)  or  incorrect  (no-go). 
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GROUP  MANAGEMENT  PLAN;  Arrangement  whereby  instruction  is  scheduled 
and  conducted  for  groups  instead  of  individuals. 


GROUP  TRAINING:  A group  of  people  gathered  together  for  the  purpose 
of  receiving  information  or  instruction  in  \;he  performance 
of  sane  specific  task. 


HARD  DATA:  A direct  and  precise  measure  of  a specific  performance. 

A JPM  is  an  example  of  hard  data  while  an  attitude  question- 
naire is  a less  direct  measure,  providing  soft  data. 

HIGH  DENSITY  SIGNAL:  A signal  containing  many  cues.  A low  density 
signal  contains  few  cues. 


INDEPENDENT  RELATIONSHIP;  Occurs  when  skills  and  knowledges  in  one 
objective  are  un*^elated  to  those  in  the  other  objective. 
Mastering  one  of  the  objectives  does  not  simplify  the 
other. 


INDICATOR  SEKAVIOR:  ilefers  to  that  behavior  that  indicates  the 
presence  of  a specific  attitude. 


INDIVIDUALIZED  INSTRUCTION:  Refers,  in  the  ISD  Model,  to  a management 
scheme  which  permits  individual  characteristics  of 
trainees  to  be  a major  deteminant  of  the  kind  and  amount 
of  instruction  given.  Here,  it  nearly  always  implies 
some  form  of  self-pacing. 


INSTALLATION  SUPPORT  SCHOOLS:  Organized  and  operated  by  individual 
units  or  commands  to  meet  local  training  requirements. 


INSTRUCTIONAL  CONDITIONS:  The  amount  of  participation  which  the 
instruction  requires  of  the  learner.  Instructional 
conditions  may  be  active  (the  learner  produces  or  prac- 
tices) or  passive  (the  learner  sits  and  listens). 


INSTRUCTIONAL  DESIGNER:  Person  who  designs  and  develops  a program  or 

course  of  studies  based  on  a systematic  analysis. 


INFORMATION:  Knowledge;  the  facts,  names,  labels,  and  larger  bodies 

of  knowledge  that  are  necessary  for  successful  job 
performance. 
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INSTRUCTIONAL  MANAGEMENT  PLAN:  The  specifications  for  the  scheduling 
instruction  and  evaluation  of  trainees  toward  the  goal  of 
course  conpletion. 

INSTRUCTIONAL  PROGRAM:  The  develo;xnent  of  various  materials  (books, 
audiovisual  productions,  etc.)  designed  to  achieve  a 
specific  training  goal. 


INSTRUCTIONAL  SETTING;  The  vehicle  through  which  a trainee  who 

initially  is  not  able  to  perform  a task  becomes  profi- 
cient in  performing  the  task;  for  example,  performance 
aids,  self-teaching  exportable  packages,  formal  cn-job 
training,  installation  support  schools,  and  resident 
schools. 


INSTRUCTIONAL  SUPPORT:  Learning  resources;  different  kinds  of 

material,  number  of  instructors,  amount  of  time,  etc. 
which  will  contribute  to  the  learning  situation. 


INSTRUCTIONAL  SYSTEM:  The  total  effort,  distinct  from  the  operating 
system  by  location,  authority,  or  mission,  that  is  con- 
cerned with  the  preparation  of  individuals  to  serve  the 
operating  system. 


INTERNAL  CUES:  Internal  biological  signals  that  initiate  or  guide 

behavior. 


INTERNAL  EVALUATION:  Assessment  of  the  effectiveness  of  an  instruc- 
tional program  in  terms  of  student  performance  on 
stated  terminal  learning  objectives. 


JOB:  The  duties  and  tasks  performed  by  a single  worker  constitute 
his  job.  If  identical  duties  and  tasks  are  performed  by 
several  individuals,  they  all  hold  the  same  job.  The 
job  is  the  basic  unit  used  in  carrying  out  the  personnel 
actions  of  selection,  training,  classification,  and 
assignment. 


JOB  ANALYSIS:  The  basic  method  used  to  obtain  a detailed  listings  of 

duites,  tasks,  and  elements  necessary  to  perform  a 
clearly  defined,  specific  job,  involving  observations  of 
workers  and  conversations  with  those  who  know  the  job, 
in  order  to  describe  in  detail  the  work  involved,  includ- 
ing conditions  and  standards. 
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JOB  FIDELITY:  The  degree  to  which  a testing  situation  truthfully 
and  accurately  reflects  the  job  situation. 


JOB  PERFORMANCE  MEASURES:  Tests  that  are  used  to  evaluate  proficiency 
of  a job  holder  on  each  task  he  performs. 


JOB  PERFORMANCE  TEST:  Test  used  to  determine  whether  or  how  well  an 
individual  can  perform  a job.  In  may  include  either  all 
of  the  job  performance  measures  for  a particular  job  or 
a subset  of  the  job  performance  measures. 


JPA— JOB  PERFORMANCE  AID:  A cnecklist,  instruction  sheet,  or  other 
device  that  offers  a possible  alternative  to  training 
rather  than  an  actual  method  of  training;  they  are  de- 
veloped to  eliminate  or  minimize  training  requirements 
for  some  tasks. 


KNOWLEDGE  OF  RESULTS:  Feedback;  information  provided  to  the  student 

indicating  the  correctness  of  !iis  response.  Evaluative 
knowledge  of  results  indicates  what  a student  is  doing 
right  and  what  he  is  doing  wrong.  Comparative  knowledge 
of  results  indicates  how  the  student's  response  compares 
to  the  objective  or  standard  established  by  the  instruc- 
tor. 


LEARNER  CHARACTER! CT ICS:  The  traits  possessed  by  learners  that  could 
affect  their  ability  to  learn  (e.g.,  age,  I.Q.,  reading 
level , etc.) . 


LEARNING  ACTIVITY;  The  specific  behaviors  a student  performs  during 
a particular  episode  of  learning. 


LEARNING  ANALYSIS:  A procedure  to  identify  subelements  that  must  be 

learned  before  a person  can  achieve  mastery  of  the 
performance. 


LEARNING  CATEGORY:  A division  of  learning  behavior.  All  learning 
may  be  classified  into  one  of  four  learning  categories: 
mental  skill,  physical  skill,  information,  or  attitude. 


LEARNING  EVENT:  The  immediate  outcome  of  a learning  activity. 
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LEARNING  GUIDELINES:  Statenents  which  specify  the  learning  events 
and  activities  appropriate  to  specific  instruction. 
Learning  guidelines  combine  to  form  learning  sub- 
categories. 


LEARNING  HIERARCHY:  Graphically  portrays  the  relationships  among 
learning  tasks  in  which  some  tasks  must  be  mastered 
before  others  cun  be  learned. 


LEARNING  OBJECTIVE:  Describes  precisely  what  is  to  be  learned  in 

terms  of  the  expected  student  performance  under  speci- 
fied conditions  to  accepted  standards.  These  learning 
objectives  identify  the  mental  skills,  information, 
attitudes,  or  physical  skills  tint  are  required  to  per- 
form the  terminal  learning  objective. 


LEARNING  RESOURCE  CENTER:  Library  containing  instructional  materials 
and  areas  far  viewing  and  study. 


LEARNING  STEP:  Occurs  when  learning  objectives  are  broken  down  into 
smaller  parts. 


LEARNING  SUB-CATEGORY;  A division  of  a learning  category. 


LEARNING  TASK  ANALYSIS:  Procedure  used  in  the  domain  of  intellectual 

skills  to  identify  prerequisite  tasks  that  must  be 
learned  before  a person  can  learn  a given  task. 


LINK  TRAINER:  Mechanical  training  device  which  simulates  the  cock- 
pit of  an  aircraft. 


RESPONSE  BIAS:  Tendency  to  favor  a certain  response  over  others. 


J 


MANAGEMENT  PLAN:  Program  for  the  assignment,  monitoring,  and  assess- 

ment of  the  personnel,  materials,  and  resources  dedi- 
cated to  a specific  mission,  operation,  or  function. 


MASTERY:  In  terms  of  learning,  refers  to  meeting  all  of  the  specified 

minimum  requirements  for  a specific  performance.  Criteria 
for  mastery  are  defined  in  the  design  phase  of  the  ISO 
Model . 


142 


m 


(CAN:  Aritiwietic  average  calculated  by  adding  up  all  scores  and 
dividing  by  the  number  of  scores. 


MEASUREMENT:  Consists  of  rules  for  assigning  numbers  to  objects  to 
represent  guant^"  es  of  attributes. 

MEASUREMENT  ERRORS:  Incorrect  procedures  carried  out  during  the 

nKasuren«nt  process  which  Invalidate  the  results.  These 
errors  result  from  unfounded  assumtions  made  by  judges 
or  raters. 


MEASUREMENT  PROCESS:  The  operations  Involved  in  determining  the 

amount  of  an  attribute  (e.g.,  skill,  knowledge,  or  atti- 
tude) possessed  by  a student. 


MEDIA:  Means  for  presenting  instructional  material  to  learners:  for 
example,  bocks,  audiotapes,  and  filmstrips. 


MEDIA  ALTERNATIVE:  A form  of  instructional  material  that  contains 
the  stimulus  criteria  required  by  a specific  learning 
activity. 


^EDIA  MIX:  Combination  of  different  media  used  to  present  a unit  of 
instruction. 


MEDIA  POOL:  All  of  the  media  options  suitable  for  a given  unit  of 
instruction.  The  final  media  choice  is  drawn  from  the 
media  pool . 


MEDIA  SELECTION:  Is  the  major  means  of  determining  how  instruction  is 

to  be  packaged  and  presented  to  the  student. 

MENTAL  SET:  A preparatory  mental  adjustment,  or  readiness,  for  a 
particular  type  of  experience. 

MENTAL  SKILLS:  Those  processes  of  identifying,  classifying,  using 

rules,  and  solving  problems  that  involve  active  mental 
processing.  Mental  skills  imply  the  capability  of 
applying  the  learning  to  some  situation  and  demonstrating 
tlie  mental  skill,  such  as  thinking,  creating,  and 
analyzing. 
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WlEMONICS;  Methods  which  make  information  easier  to  renember;  memory 
aids. 


NODE  OF  INSTRUCTION:  Method  of  scheduling  materials  presentation.  The 
instructional  mode  may  be  individualized  (self-pacing)  or 
group  (block  scheduling). 


MODULE:  An  individualized  self- instructional  package  usually  con- 
taining all  the  necessary  materials  a learner  needs  to 
meet  some  or  part  of  a terminal  learning  objective. 


MULTIMEDIA  PhCKAGE:  Self-contained  instructional  unit  in  more  than 
one  medium. 


NARRATION:  Is  the  voice  overheard  on  an  audiovisual  program. 


NARRATOR:  Is  the  person  whose  voice  is  heard  describing  or  commenting 

upon  the  content  of  a film,  television  program,  etc. 


NUMERICAL  SCALE:  Measurement  device  which  associates  verbal  descrip- 
tions of  social  objects  with  numbers  and  requires  students 
to  indicate  their  attitudes  by  marking  the  appropriate 
number. 


OBSERVATION  INTERVIEW:  Job  holder  is  observed  in  the  job  environment 

performing  all  or  a substantial  part  of  the  job;  the  job 
holder  performs  the  job  while  the  analyst  ask  questions. 


OFF-LINE:  Refers  to  any  activity  which  does  not  take  place  as  part 
of  the  regular  production  process. 


OVERLEARNING:  Refers  to  the  continual  practice  on  a learning  task  by  a 

person  who  has  correctly  performed  the  task. 


PEER  TUTORING:  A form  of  instruction  in  which  students  at  the  same 
or  more  advanced  level  of  knowledge  provide  instruction 
to  students  at  the  same  or  lower  level  of  knowledge  on 
the  specific  objectives  under  consideration.  Peer  tutors 
are  not  members  of  the  existing  instructional  establish- 
ment. 
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PERFORMANCE  EVALUATION:  The  gathering  of  data  to  specifically 

determine  the  success  of  students  on  a specific  task, 
as  a result  of  a training  program. 


PERFORMANCE  MEASURES:  The  absolute  standard  by  v;hich  a job  perfor- 
mance is  judged.  A performance  measure  Is  the  inven- 
tory of  job  tasks  with  each  performance  objective. 


PERSE  ERATE:  Continue  an  activity  '.intll  it  is  completed,  regardless 
of  the  difficulty,  or  the  appropriateness  of  the  solu- 
tion technique  to  the  problem. 


PERT— PROGRAM  EVALUATION  REVIEW  TECHNIQUE:  PERT  is  a method  of  moni- 

toring the  flow  of  a large  project  by  breaking  it  down 
into  small  individual  activities  and  assigning  each 
activity  a specified  amount  of  time  for  completion. 


PHYSICAL  SKILLS:  Specified  muscular  activities  for  accomplishing 

a goal . 


POST  FEEDBACK  DELAY:  The  pause  which  follows  the  presentation  of 
feedback.  This  allows  time  for  the  correct  response  to 
"sink  in." 


POSTTFST:  A test  administered  after  the  completion  of  instruction  to 
assess  whether  a student  has  mastered  the  objectives  of 
the  course  or  unit. 


PREDICTIVE  VALIDITY:  The  ability  of  a test  score  to  accurately  fore- 
cast future  performance. 


PREDlFFEhENTIATION  OF  STIMULI:  Pointing  out  the  distinguishing 

features  of  an  object  and  explaining  the  differences 
between  them. 


PRETEST:  Administered  prior  to  instruction  to  determine  how  much 

the  student  already  knows. 


PROCESS  EVALUATION:  An  early  stage  in  ISD  development  that  identifies 
which  steps  in  the  model  wi’l  ;e  used  for  the  course  under 
development.  The  purpose  t-he  process  evaluation  is  to 
describe  and  document  the  actual  developmental  process  for 
this  particular  instruction. 
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PROCESS  STANDARDS:  Refers  to  the  conditions  which  must  be  satisfied  for 
a job  to  be  successfully  conpleted.  Process  standards  refer 
to  sequence,  accuracy,  speed  of  performance,  and  complete- 
ness. 


PR(XiRAffED  INSliUJCTK^:  Instructional  materials  which  present  subject 

matter  in  a series  of  small  sequential  units  which  require 
responses  fran  the  r.tudent. 


PRWiPr:  A word  or  other  signal  that  initiates  or  guides  behavior; 
a cue. 


QUALITV  COfiTROL:  Process  of  measuring  and  evaluating  in  order  to  main- 
tain course  standards  through  adjustment  in  instructional 
materials  or  procedures. 

QUALITY  CONTROL  DATA;  Information  which  reflects  the  degree  of  success 
achieved  by  a system  or  operation . 


RANDOM  SELECTION:  Choosing  people  or  objects  at  random  rather  than 
according  to  some  systematic  plan. 


RANK  ORDER:  The  assigrunent  of  ranks  to  students.  This  could  refer  to 

groups,  such  as  the  top  10%,  or  simply  listing  each  stu- 
dent from  highest  to  lowest.  Hank  ordering  is  appro- 
priate when  there  is  a need  to  select  the  fastest,  the 
most  accurate,  or  the  >>ist  producer. 


RATING  ERRORS:  Errors  of  standards,  ratio,  and  logic. 


RATING  SCALE:  A messuronent  device  in  which  a student  must  choose  a 
response  from  a range  of  choices  arranged  in  a continuum 
from  low  to  high  or  good  to  bad,  etc 


REGULATIONS:  Rules  for  appropriate  conduct  and  behavior. 


RELIABILITY:  The  consistency  with  which  a test  measures  the  amount  of 

student  achievement. 


RESIDENT  SCHOOLS:  These  schools  are  designed  to  meet  service-wide 

training  requirements. 
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REVISION  PLAN:  A detailed  outline  of  the  procedures  to  be  taken 

to  modify  the  structure  or  content  of  a course. 


REWARD  SEQUENCE:  Scheduling  the  more  pleasant  activity  to  follow 

the  less  pleasant  activity;  can  be  used  to  provide 
a reward  for  completion. 


SAMPLE:  A portion  or  small  segment  of  the  students  for  whom  instruc- 

tion is  designed. 


SAMPLING  PLAN:  Procedure  for  selecting  a small  but  representative 

group  from  a larger  population. 


SCALE:  In  media  selection,  some  materials  must  represent  actual 

objects  and  accurately  represent  the  dimensions  of 
those  objects.  A model  may,  for  example,  be  full 
scale,  half  scale,  or  on  a 1 to  10  scale  with  the 
actual  object. 


SELF  PACING;  Mode  of  instruction  whereby  each  student  works  through 
the  instructional  materials  at  his  own  rate  of  speed. 


SELF-PACED  MANAGEMENT  PLAN:  Arrangement  whereby  instruction  is 

scheduled  and  conducted  for  individual  students  rather 
than  groups  of  students. 


SELF-TEACHING  EXPORTABLE  PACKAGES:  Self  instructional  study  units; 

generally  sent  to  the  student  wherever  he  is  stationed. 


SEQUENCING:  Ordering  instruction;  proper  sequencing  allows  the 

learner  to  make  the  transition  from  one  skill  or  body 
of  knowledge  to  another,  and  assures  that  supporting 
skills  and  knowledge  are  acquired  before  dependent 
performances  are  introduced. 


SHAPING:  Gradually  changing  a student's  behavior  until  it  is  correct 

SIGNAL:  Cue  that  initiates  and  directs  activity. 


SIMULATION: 
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Any  change  from  reality  or  any  imitation  of  reality. 
Three  types  are  common:  simulating  part  of  the  system, 

simulating  the  operation  of  the  systan,  and  simulating 
the  environment  in  which  the  system  will  operate. 


SIMULATORS;  Mcchines  or  processes  designed  to  provide  training  which 
will  have  high  positive  transfer  to  the  real  world  equip- 
ment or  situation.  Simulators  are  ordinarily  cheaper, 
safer,  or  more  available  than  thu^  actual  situation  or 
equipment. 


SLIDE-TAPE;  A ■combination  of  visual  slides  and  an  audio  tape  syn- 
chronized so  that  the  audio  describes  the  content  of 
the  slides. 


SOFT  DATA:  Obtained  from  attitude  or  opinion  surveys.  This  data  is 
not  as  reliable  as  hard  data. 


STANDARDS:  Occurs  in  terminal  learning  objectives  or  learning 

objectives;  describes  the  criterion  or  standard  of  per 
formai.ee  which  must  be  attained. 


STIMULUS  CRITERIA;  Those  c^sic  qualities  or  capabilities  of  a 

medium  that  are  required  to  carry  out  the  intent  of  the 
learning  activity;  for  example,  visual  images,  motion, 
color,  and  sound. 


STORYBOARD:  A collection  or  series  of  small  pictures  which  describe 

the  action  and  content  that  will  be  contained  in  an  audio- 
visual or  visual-only  production.  A sequence  of  these 
small  pictures  comprise  a storyboard. 


SUBJECT  fWTER  EXPERT;  A person  who  has  professional  skill  in  the 
performance  of  some  job  and  who  is  consulted  by  an  in- 
structional designer  in  the  process  of  job  task  analysis. 


SUPPORTIVE  RELATIONSHIP:  Occurs  when  skills  and  knowledges  in  one 

objective  have  some  relationship  to  those  in  the  other 
objective;  the  learning  involved  in  mastery  of  one  learn- 
ing objective  transfers  to  the  other,  making  learning 
involved  in  the  mastery  of  the  other  easier. 
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SYMBOL:  Anything  that  stands  for  or  represents  something  else.  A 
plus  sign  (+)  is  a symbol  for  the  mathematical  operation 
of  addition. 


SYSTEM  (4ASTER  PLAN:  Control  document  used  to  coordinate  the  develop- 

ment and  implementation  of  an  Instructional  program. 


SYNCHRONIZING  PULSE:  An  audible  or  inaudible  sound  used  to  coordinate 
the  audio  and  video  portions  of  a slide-tape  program  so 
that  audio  and  video  (i.e.,  slide  and  narration)  ar.,- 
coordinated. 


SYSTEMS  APPROACH:  A generic  term  referring  to  the  orderly  process  of 

analysis,  design,  development,  evaluation,  revision,  and 
operation  of  a collection  of  interrelated  elements. 


TALK-THROUGH  TECHNIQUE:  Occurs  during  tho  simulation  of  an  operational 

system;  involves  talking  through  each  operation  in  the 
new  system  to  detemine  decisions  and  contingencies. 


TARGET  POPULATION:  The  pool  of  potential  entrants  to  training  for 

which  instructional  materials  are  designed  and  tried  out. 


TASK  DELAY  TOLERANCE:  A measure  of  how  much  delay  can  be  tolerated 
between  the  time  the  need  for  task  perfovmance  becomes 
evident  and  the  time  actual  perfmmance  must  begin. 


TASK:  Formed  in  clusters  which  make  up  duties.  A task  is  the  lowest 

level  of  behavior  In  a job  that  describes  the  performance 
of  a meaningful  function  in  the  job  under  consideration. 


TASK  INVENTORY:  List  that  itemizes  all  of  the  tasks  that  make  up 

a selected  duty. 


TASK  LEARNING  DIFFICULTY:  Refers  to  time,  effort,  and  assistance  ‘re- 

quired by  a student  to  achieve  performance  proficiency. 


TASK  STANDARD;  A statement  of  how  well  a task  must  be  pe» formed. 
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TASK  STATEMENT:  A statenent  of  highly  specific  action  which  has  a 
verb  tjnd  object;  for  example,  sort  mall. 


TECHNICAL  ORDERS:  Military  regulations  which  deal  with  the  specific 
nature  of  technical  materials  and  equipment. 


TERMINAL  LEARNING  OBJECTIVE:  bet  ived  from  job  performance  measures, 
TLOs  are  to  be  attained  during  training.  Ti.Os  are  broken 
down  into  their  component  parts  which  are  documented  as 
learning  objectives  which  may  be  further  divided  into 
learning  steps.  Each  TLO  contains  actions,  conditions, 
and  standards. 


TESTS:  Any  device  or  technique  used  to  measure  the  pe.'formance  of  a 

student  on  a specific  task  or  subject  matter. 


TESTING  CONSTRAINTS:  Limitations  such  as  time,  ^noney,  personnel, 

facilities,  and  other  resources,  which  prohibit  job 
performance  measures  from  being  identiccl  to  the  tasks 
they  measure. 


TRADE-OFFS;  in  any  systematic  approach  to  instruction,  it  is 

necessary  to  make  compromises  bet*  een  what  is  desirable 
and  what  is  possible.  Ordinarily,  these  decisions  in- 
volve increases  or  decreases  in  time,  money,  facilities, 
equipment,  or  personnel.  Training  aids  and  simulators 
represent  examples  of  trade-offs. 


TRAINER  APPRAISAL  KIT:  A package  of  instructional  materials 

designed  to  provide  a course  instructor  with 
practice  in  the  preparation,  presentation,  and  valida- 
tion of  instruction. 


TRAINING:  The  teaching  of  job  skills.  It  can  take  a number  of  forms 

such  as  self-teaching  exportable  packages,  training 
manuals,  individual  learning  packages,  FOJT,  or  group 
training. 


TRAINING  SETTING  CRITERIA:  In  media  selection,  the  options  that 

training  must  be  either  small  group,  large  group,  indi- 
vidualized at  a fixed  location,  or  individualized  inde- 
pendent of  location. 
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TRYOUT:  Practice  test;  the  purpose  is  to  make  the  tryout  as  realistic 
as  possible  by  eliminating  as  many  sources  of  unreliability 
as  possible. 


UNDERTRAIN:  Provide  Inadequate  training  that  does  not  prepare  a 

student  to  meet  regular  job  performance  requirements. 


VALIDATION:  A process  through  which  a course  is  revised  until  it 
is  effective  in  realizing  its  instructional  goal. 


VALIDATION  DOCUMENTATION:  A report  which  describes  in  detail  how 
a specific  course  of  instruction  was  validated  and  for 
what  target  population. 


VALIDATION  PROCESS:  Testing  instructional  materials  on  a sample  of 
the  target  population  to  insure  that  the  materials  are 
effective. 


VALIDITY:  The  degree  to  which  a test  measures  what  it  claims  to 

measure. 


VALUE  ENGINEERING:  Refers  to  the  process  of  designing  equipment 

or  instruction  to  meet  but  not  exceed  the  required  out- 
comes. Ordinarily,  it  refers  to  the  elimination  of 
features  or  instructional  objectives  that  have  not  been 
demonstrated  to  be  positively  necessary. 


VIGILANCE  LEVEL.:  General  degree  of  watchfulness  or  attentiveness 

to  what  may  come. 


VISUAL  FORM:  In  media  selection,  refers  to  whether  alphanumeric  or 

pictorial  characteristics  are  required  in  a learning 
situation. 


VISUAL  SPECTRUM:  The  type  of  color  required  of  instructional 

materials.  Some  must  be  with  full  color,  others  may 
be  with  black  and  white  or  shades  of  grey. 


WITHIN-COURSE  TESTS:  Administered  during  a course  of  instruction 

to  assure  that  all  students  are  "keeping  up"  with  the 
learning  objectives. 


151 

WORK  ELEMi".NTS:  The  element  is  the  smallest  cwnponent  in  the 

structure  of  a job.  Elements  canbine  to  form  a task* 
tar.ks  combine  to  form  a duty,  and  duties  combine  to 
form  a job. 


Needing  page  blank  indi^x 


NOTE:  The  Roman  numerals  folio’.^ing  each  entry  refer  to  the  Phase 

numbers;  the  Arabic  numerals  refer  to  the  page  numbers. 


Ability  grouping,  III:  125 
Achievement  grouping.  III:  125 
Activities,  see  Learning  activities 
Adjunct  programs,  III;  251 
Affective  component:  attitude 
learning.  Ill:  66-67,  103 
Air  Force  Specialties  (AFS),  I:  3,8 
Algorithms,  III:  20-22,  71-104 
Antisubmarine  warfare  (ASW) 

training  program,  Navy,  III:  142 
Attitude  U'arning:  learning  sub- 
category  11,  III:  66-70,  103-104 
affective  component,  III:  66-67, 
103 

behavioral  component.  III:  66. 
70,  104 

cognitive  component.  III:  66, 
67-70,  104 

Audio  producer  (production  person- 
nel). Ill:  228 

Audio-visual  production,  III:  235- 
244 


Baseline  data  collection,  V:  25-26, 
70-71 

Behavioral  component:  attitude 
learning,  UI:  66,  70,  104 
Block  scheduling  (group  instruc- 
tion plan).  III;  124-125 


Classifying:  learning  sub- 
category 2,  III:  38-40,  74-75 
Clustering,  task,  I:  244-245 
Cognitive  component:  attitude 
learning.  III:  66,  67-70,  104 
Command  Job  Analysis,  schedule  for, 
I:  58 

Cotmion-factor  learning  objectives, 
II:  90-92 

Comprehensive  Occupational  Data 
Analysis  Program  (CODAP), 

I:  121 


Computer-assisted  instruction  (CAI), 
HI:  191 

Computer-managed  instruction,  III: 
191-194 

Conditions,  JPM,  I;  177-173 

on-the-job,  task,  I:  24-26,  89-93 
Contiguity,  III:  41 
Conti ngency-manay ed  i ns tructi on , 

III:  187-189,  215 

Contract,  student  (or  performance), 
III:  189 

Cues.  JPM,  1:  179 

prompts  for  student  learning.  III: 
6*8 

task,  I:  25,  27-28 


Data  collection, 

for  external  evaluation,  V:  68-77 
for  internal  evaluation,  V:  1-8, 
29-46,  53-62 
for  JPMs,  1:45-46 
for  rating  tasks,  I:  133-139 
Data  collection  plan  for  job  analysis, 
alternate  or  short-cut,  T:  47, 

77-78 

form  preparation,  I:  49-51,  59 
group  interviews,  I:  40 
individual  and  observation  inter- 
views, I:  36-37,  41-46  52-62 
jury-of-experts,  I:  39-40 
questionnaire  survey,  I:  37-39, 

45,  46,  68-73 

Decision-making:  learning  sub-category 
5,  III:  46-48,  83-86 
Decisions,  management.  Executive 
Sumnary:  20,  24-25,  3G^3TT3^-37. 
42-437~Sl , 54-55,  59,  63,  69-70, 
75-76,  81-82,  85-86,  91-92,  96-97, 
100-101,  108-109,  114-115,  119-120 
Decision  trees,  see  Algotithms 
Defense  Occupational  Specialties  (DOS), 
I:  3,  8,  214 

Delay  tolerance,  s^  Task  delay  ... 
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Formative  eva'!uation.  III:  280-281 
Front  End  Analysis,  I:  216-218 


Graphic  artist  (production  person- 
nel ).  Ill:  227-228 


Hard  data,  V:  4 


Identifying  symbols,  learning  sub- 
category 3;  III:  41-42,  76-77 
Installation  Support  Schools  (ISS), 
I:  238,  258-259;  III:  140-142 
Instruction,  conducting,  IV:  29-39 
course  documentation,  pre- 
instruction  review  of,  IV: 
32-34 

changes  in,  IV:  36-38 
documentation  of  changes, 
problems  for,  IV:  35-37 
follow-up  activities,  IV:  37-38 
function  of  instructor,  IV:  29- 
31,  32-38 
outputs,  IV:  38 
procedures,  IV:  32-38 
test  administration,  IV:  32-33, 
34-35 

training  of  instructors,  IV:  34- 
35 

Ir.struction,  development  of.  III: 
221-287 

adjunct  programs.  III:  251 
aims,  lit:  221-223 
audio-only  scriot  guidelines, 
III:  230-232,  233 
audio-visual  production.  III: 
235-244 

Formal  On-the-Job  Training 
(FOOT),  III:  260-262 
Instructor's  Guide,  III  265- 
266 

Job  Performance  Aias  (JPAs), 

III:  252-260 

needs  and  constraints.  III: 
224-225 

outputs.  III:  270 
platform  lectures.  III:  248 
pre-testing  of  draft  materials, 
III:  263-265 

printed  materials,  III:  246 


Instruction,  development  of  (con- 
tinued) 

proceoures.  III:  224-269 
production  personnel.  III:  227-229 
programmed  instruction,  III:  246- 
247 


resources.  III:  225-230 
Self-Teaching  Exportable  Packages 
(steps).  III:  248-249 
slide-tape  production.  III:  239- 
244 

Students'  Guide,  III:  269 
supplementary  instruction.  III: 
249-250 

television  program  production, 
III:  244-246 

cerms  defined.  III:  286-287 
tyoes  of  instruction.  III:  230- 
263 

user  instructions.  III:  265 
video-only  materials.  III:  232, 
234-235 

Instructional  inanagement  plan  and 
delivery  system,  III:  105-197 
block-scheduling  for  group 
instruction.  III:  124-125 
consumables  and  courseware.  III: 
136-137 

course  management,  III:  126-127 
definition.  III:  105 
equipment  and  facilities,  III: 
135-136 

for  instructors,  III:  131-135 
for  student  processing.  III:  127- 
130 

for  support  personnel.  III:  135 
function  of  guidelines.  III:  '06- 
107 

group  assignment,  methods  of, 

III:  125 


management  guidelines.  III:  12A- 
137 

media  costs,  III:  112 
media  mixes.  III;  T 07-1 18 
media  selection,  III:  118-124 
outputs.  III:  142-143 
procedures  for  specifying,  III: 
107-142 

program  completion,  III;  131 
self-pacing  mode,  III;  l'^4,  125- 
126 

System  Master  Plan  (SMPl,  III: 
105,  107,  138-142 
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Formative  evaluation.  III:  280-281 
Front  End  Analysis*  I:  216-218 


Graphic  artist  (production  person- 
nel), III:  227-228 


Hard  data,  V:  4 


Identifying  symbols,  learning  sub- 
category 3;  III:  41-42,  76-77 
Installation  Support  Schools  (ISS), 
I:  238,  258-259;  III:  140-142 
Instruction,  conducting,  IV:  29-39 
course  documentation,  pre- 
instruction  review  of,  IV: 
32-34 

changes  in,  IV:  36-38 
documentation  of  changes, 
problems  for,  IV:  35-37 
follow-up  activities,  IV:  37-38 
function  of  instructor,  IV:  29- 
31,  32-38 
outputs,  IV:  38 
procedures,  IV:  32-38 
test  administration,  IV:  32-33, 
34-35 

training  of  instructors,  IV:  34- 
35 

Instruction,  development  of.  III: 
221-287 

adjunct  programs.  III:  251 
aims.  III:  221-223 
audio-only  script  guidelines, 
III:  230-232,  233 
audio-visual  production.  III: 
235-244 

Formal  On-the-Job  Training 
(FOJT),  III:  260-262 
Instructor's  Guide,  III:  2S5- 
266 

Job  Performance  Aids  (JPAs), 

III:  252-260 

needs  and  constraints,  III: 
224-225 

outputs,  III:  270 
platform  lectures.  III;  248 
pre-testing  of  draft  materials, 
III:  263-265 

printed  materials,  III;  246 


Instruction,  development  of  (con- 
tinued) 

procedTures,  III:  224-265 
production  personnel.  III:  227-229 
progranmed  instruction.  III:  246- 
247 

resources.  III;  225-230 
Self-Teaching  Exportable  Packages 
(steps),  III:  248-249 
slide-tape  production,  III:  239- 
244 

Students'  Guide,  III:  269 
supplementary  instruction.  III: 
249-250 

television  program  production, 

III:  244-246 

terms  defined,  III:  286-287 
types  of  instruction.  III:  230- 
263 

user  instructions.  III:  265 
video-only  materials,  III:  232, 
234-235 

Instructional  management  plan  and 
delivery  system.  III:  105-197 
block-scheduling  for  group 
instruction,  III:  124-125 
consumables  and  courseware,  III: 
136-137 

course  management.  III:  126-127 
definition,  III;  105 
equipment  and  facilities,  III: 
135-136 

for  instructors,  III:  131-135 
for  student  processing,  III:  127- 
130 

for  support  personnel.  III:  135 
function  of  guidelines,  HI:  106- 
107 

group  assignment,  methods  of, 

III:  125 

management  gui del ines.  III:  124- 
137 

media  costs,  III;  112 
med^a  mixes.  Ill:  107-118 
media  selection,  III:  118-124 
outputs.  III;  142-143 
procedures  for  specifying.  III: 
107-142 

program  completion.  III:  131 
self -pacing  mode,  III:  124,  125- 
126 

System  Master  Plan  (SMP),  III: 

105,  107,  138-142 
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Instructor  evalnation  program,  IV: 
7-8 

Instructor-managed  instruction.  III: 
186-U7 

Instructor(s ) , role  of,  IV:  7-9, 
29-31,  35-38 
training  of,  IV:  7-12 
Instructor's  Guide,  III:  265-263 
Instructor's  Manual,  IV:  4-5 
Internal  evaluation,  IV:  2-4,  5,  9; 
V:  1-62 

conducting  process,  V:  33-34 
data  forms  and  questionnaires 
(examples),  V:  53-62 
data  rating  and  measurement,  V; 
1-8 

data  (general)  display  and  in- 
terpretation, V:  34-40 
data  sources  and  uses,  V:  5-8 
documentation,  V:  49-52 
evaluation  designs  (additional), 
V;  46-49 

functions  and  requirements  uf, 

V:  1-10 

ISO  documentation  for  process 
evaluation,  V:  17-19 
outputs,  V:  49 

performance  data  display  and 
interpretation,  V:  40-46 
Performance  Evaluation  Plan,  V: 
22-28 

Plan  for  Collecting  Information 
from  Instructors,  V:  31-33 
Plan  for  Collecting  Information 
from  Suudents,  V:  29-30 
plan  (overall)  development,  V: 
11-33 

procedures,  V;  11-49 
Process  Evaluation  Plan,  V:  14- 
22 

product  evaluation,  V:  20 
Program  Evaluation  Review  iech- 
nique  (PERT),  V:  10 
Progress  Ev-iluatior.  Plan,  V:  11- 
14 

project  schedule,  V:  10 
repori  (INER),  V:  34,  49 
sample  summary  of  internal  eval- 
uation report,  V:  50 
Internal  Evaluation  Report  (INER), 

V:  34,  49,  90,  93,  95,  105 
Intcrview(s)  for  data  collection, 
arif  angements  For,  IV:  54 


Interview(s) , (conti nued) 

briefings  preceHTngT  I:  55-57 
criteria  for  interviev/ee  selection, 
I:  52-54,  59 

for  external  evaluation,  V:  74-75 
forms,  I:  49-51,  59,  134-136 
group,  I:  40 

guidelines  for  conducting,  I:  59- 
61 

Individual,  on-site,  observation, 

I:  36-37,  41-46,  52-54 
schedule  for  conducting,  I:  58 


Job,  I:  8 

Job  analysis,  I : 1-112 

daily  schedule  for,  I;  58 
data  collection,  I:  34,  36-78 
data  revision,  I:  74-76 
documentation  for,  J:  83-85 
functions  of,  I:  2,  5 
interviews  for,  I:  36-37,  40-46, 
52-61 

job  analysts  (personnel),  I:  48- 
49 

of  equipment-oriented  jobs,  I:  81- 
83 

of  new  jobs,  I:  41,  78-83 
of  non-equipment-oriented  jobs,  I: 
80-81 

outputs,  I:  33-34,  85 
overview  of,  I:  33-34 
procedures,  I:  34-83 
questionnaire  surveys  for,  I:  37 
39.  44-46,  68-73 

task  inventory  validating,  I:  68 
task  inventory  verifying,  I:  66-67 
terms  defined,  I:  6-33 
Job  data  worksheets,  samples,  I:  51, 
63-65,  192-193,  210 
Job  Performance  Aids  (JPAs),  I:  18, 
114,  231  . 233-235,  245-249-, 

III:  139,  252-260 
Job  performance  evaluation  data 
collection,  V:  71-73 
Job  Performance  l.easures  (JPMs), 
in  analyzing  existing  courses,  I: 
216,  225-226 

in  developing  learning  objectives, 
II:  2,  6 

in  external  evaluation,  V:  70-73, 
80 

in  internal  evaluation,  V:  43 
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Job  Perforniance  Measures  (con- 
ti nued ) 

in  selecting  instructional 
settings,  I:  232,  242 
in  system  revision,  V;  110-111 
In  testing,  II:  44 
Job  Perfonnance  Measures,  con- 
struction of,  I:  156-212 
checklists  for,  I:  196-197,  211 
conditions,  I:  177-178 
cues.  I;  179 

cut-nff  scores  for,  1;  200-201 
data  collection  for,  I:  45,  46 
documentation  for,  I*  191-193, 
209 

errors  in  Measurement,  I;  201- 
205 

outputs,  I:  209 
part-task  testing,  I;  185-187 
physical  fidelity,  I:  161-163, 
167 

predictive  validity,  I:  158-16'i 
167 

procedures  for,  I:  165-209 
process  and  product  rating,  I: 
170-171,  195-196 
rationale  for,  I:  156-157 
rating  scales  for,  I:  197-200 
sample  list  of  validated  JPMs, 
I:  210 

sampling  plan,  I:  187-191 
scoring  procedures,  I:  191, 
194-204 

simulator  requirements  for,  I: 
163,  172-176 
standards,  1:  180-184 
testing  constraints,  I:  166- 
170 

tryout  procedure,  I:  207-208 
types  of  tasks  measured,  I; 
164-165 

validation  and  revision,  I: 
205-209 

Job  Performance  Test  (OPT),  I:  158 
Jury-of-Expcrts , I:  39-40,  41,  67 


Knowledge  of  results  (KOR)r 
31,  32,  33,  48,  51 


III. 


Learning  activities,  Learning 
events/activities,  specifying 


Learning  analysis,  II:  18-30 
attitude  category,  II;  28-30 
Infrrmation  category,  II:  23-26 
mental  skills  category,  II:  19-23 
physical  skills  category,  II:  26- 
28 

Learning  categories  and  sub-categories, 
II:  16-17,  III:  11-17 
Learning  events/activities,  specifying, 
III:  1-105 

classification  of  learning 
objectives.  III:  10-17 
conditions  (active  vs.  passive), 

III:  4-5 

cues  or  prompts.  III:  6-8 
directions  for.  III:  24-26 
feedback,  III:  6,  9-10 
flowcharts  (algorithms).  III:  18, 

20.  71-104 

guidelines  for  eleven  sub- 
categcries  of  learning.  III; 
17-18,  35-70 

learning  categories  with  sub- 
categories, III:  11-17 
learning  guidelines  (general).  III: 
3-6 

outputs,  III;  26 
procedures,  III:  2-25 
purposes,  III:  1-3 
Learning  guidelines,  functions  of, 

III:  1-4,  208-213 
general,  III:  3-6,  10 
specific,  for  eleven  learning 
sub-categories.  III:  17-20, 

35-70 

sub-category  1:  rule-learning  and 
using.  III:  35-37,  I’1-73 
sub-category  2:  classifying,  III: 
33-40,  74-75 

sub-category  3:  identifying  symbols, 
III:  41-42,  76-77 
sub-category  4:  detecting,  HI: 
42-45,  78-82 

sub-category  5:  making  decisions, 
III:  46-48,  83-86 

sub-category  6:  recalling  bodies  of 
knowledge.  III:  49-52.  87-90 
sub-category  7:  performing  aross 
motor  skills.  III:  53-55,  91-93 
sub-category  8;  steering  and  guid- 
ing, III:  56-57,  94-95 
sub-category  9:  positioning  move- 
ment, III;  58-62,  96-99 
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Learning  guidelines  (continued) 
sub-category  10:  voice  coiimuni- 
cating.  III:  63-65,  100-102 
sub-category  11:  attitude 
learning.  III:  66-70,  103- 
104 

Learning  objectives  (LO),  II:  1, 

4-5,  8-30,  see  also  Objectives, 
conmon-factor,  II;  90-92 
testing,  II:  36,  38,  39,  42-45. 
49-50,  64 

Learning  Objective  Analysis  Work- 
sheet. II:  7;  III;  27,  109,  224- 
225 

Learning  objectives,  sequencing  and 
structuring,  see  Sequence  and 
structure  learning  objectives, 
determining 

Learning  Resource  Centers  (LRCs), 
III:  141 

Learning  steps  (LSs),  II:  4,  5,  36, 
38,  57 


Management  decisions,  see  Decisions, 
management 

Management  guidelines,  III:  124-137 
Management  plan,  sm  Instructional 
management  plan  and  delivery 
system 

Management  plans,  student,  III;  185- 
196 

combination  plans.  III:  196 
computer-managed  instruction, 

HI:  191-194 

contingency-managed  instruction, 
III:  187-189 

instructor-managed  instruction, 
III:  186-187 

media-managed  instruction.  III: 
194 

peer-managed  instruction,  III: 
190-191 

student  self-managed  Instruction, 
HI:  195 

Marginal  students,  guidelines  for 
managing,  III:  129-130 
Materials,  existing— review  of, 

see  Existing  materials,  revie./  of 
Matrices  (for  media  selection).  III: 
113,  115-118,  172-184 
Media,  comparative  costs,  III:  112 


Media  (continued) 

dec IsTorTmaTr ices  for  selection 
of.  III:  172-184 
evaluation  of,  Existing 
materials,  review  of 
revision.  III:  206-208 
selection  (for  instructional 
management  plan),  III:  106- 
124,  172-184 

Media-managed  instruction.  III:  194 

Media  specialist  (production  person- 
nel), III:  227 

Military  Occupational  Specialties 
(MOS),  1:  3,  8 

Misclassi fications , in  testing,  II; 
52-55 

Mnemonics,  III;  31,  36,  41,  50,  59 


Non-equipment-oriented  jobs,  I:  79- 
81 


Objectives,  development  of,  II:  1- 
34 

action  statement,  II:  5-12 
conditions  statement,  II;  5-6, 
9-10,  13-14 

criteria  for  Terminal  Learning 
Objective  and  Learning  Objec- 
tive statements,  II:  8-16 
incuts,  II:  1-2 

Job  Performance  Measures  in,  II: 
2 

learning  analysis  for  each  Term- 
inal Learning  Objective,  II: 
18-30 

learning  analysis:  attitude 
category,  II:  28-30 
learning  analysis:  information 
category,  II:  23-26 
learning  analysis:  mental  skills 
category,  II:  19-23 
learning  analysis:  physical 
skills  category,  II:  26-28 
learning  categories,  II:  16-17 
learning  objectives  iLOs),  II: 
4-5,  8-30 

learning  steps  (LSs),  II;  4-5 
outputs,  II:  30 
procedures,  II:  6-30 
standards  statement,  II:  5-6, 
9-10,  14-16 


Objectives,  development  of  (con- 
tinued) 

temuTal  learning  objectives 
(TLOs)  preparatior , II:  2-5, 
6-16 


Peer-managed  inst> uction.  III:  190- 
191,  215 

Perceptual  set(';).  III:  46,  63 
Performance  Evaluation  P’an,  V;  22- 
28 

area  of  "entry  skills,"  V:  24- 
25 

area  of  "external  requirements," 
V:  23-24 

area  of  "tests,"  V:  25-26 
area  of  "time  required  to  com- 
plete instructional  units," 

V:  27-23 

Performing  gross  motor  skills: 
learning  sub-cateqory  7,  III: 
53-55,  91-93 

Pnotographer  (production  personnel), 
III:  228 

Physical  fidelity,  of  Job  Perform- 
ance Measures,  I:  161-163,  167 
Platform  lectures,  III:  248 
Positioning  movement  and  recalling 
procedures:  learning  sub-category 
9,  III:  58-62,  96-99 
Post-feedback  delay,  III:  61 
Posttest,  V:  109-110 
Predictive  validity  of  Job  Per- 
formance Measures,  I;  158-161, 

167 

Predifferentiation  of  stimuli.  III: 
30,  41 , 49 
Pretest,  II:  73-75 
Print  specialist  (production  per- 
sonnel), III:  228 
Process  Evaluation  Plan,  V;  14-22 
Process  rating,  I:  196 
Product  rating,  I:  195 
Program  Evaluation  Review  Tech- 
nique (PERT) , V:  10 
Programmed  instruction.  III;  246- 
247 

Progress  Evaluation  Plan,  V:  11-14 


Questionnaire  survey,  administra- 
tion of,  I:  72-73,  106-112 


Questionnaire  survey  (continued) 
closed  form,  I:  38-3^1 
for  external  evaluation  data 
collection,  V:  73-74 
for  jjb  analysis,  I:  37-39,  44- 
46 

for  rating  tasks  for  training, 
I:  133-136 
open  form,  I:  38-39 
preparation  of,  I:  68-71,  134- 
136 

sample  selection  for,  I:  71-72, 
133-134 


Random  grouping,  III:  125 
Rank-ord.,r  testing,  II:  52-55 
Ratings  (Navy),  I:  3,  8 
Rating  scales  (for  Job  Performance 
Measures),  description  of,  I: 
197-198 

graphic,  I:  198-200 
numerical,  I:  197 
Recalling  bodies  of  knowledge: 
learning  sub-category  6,  III: 
49-52,  87-90 

Reinforcement;  reinforcer,  III: 
187-189 

Resident  Schools  (RS),  I:  238- 
239,  259;  III:  139-141 
Response  biases.  III:  46 
Revision  of  system,  sou  System 
revision 

Revision  plan  format,  V:  113 
Rule-learning  and  usir.j-.  learning 
sut-categcry  1,  III:  35-37,  71-73 


Sampling  plan  for  Job  Pe'^formance 
Measures.  I:  187-191 
Self-pacing  instructional  plan,  III: 
124,  125-126 

Self-Teaching  Exportable  Packages 
(steps),  i:  231,  235-237,  250- 
255;  III:  140-142,  248-249 
Sequence  and  structure  of  learning 
objectives,  determing,  II:  79-96 
common -factor  learning  objectives 
II:  90-92 

determining  relationships,  II; 
81-82,  83 

grouping,  II:  92-94 
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Sequence  and  structure  of  learning 
objectives,  dcto>Tnininq  (contir- 
ued) 

outputs,  II:  94 
procedures,  II;  81-93 
purposes  of,  II:  79-80,  88-39 
with  dependent  relationships, 

II:  82-87 

with  independent  relationships, 
IT:  90 

with  supportive  relationships, 
II:  87-88 

■Setting,  see  Instructional  set- 
ting, selection  of 
Shaping,  III:  55,  57 
Simulator  requirements  in  Job 
Performance  Measures,  I:  163, 
172-176 

Slide-tape  production.  III:  239- 
244 

Soft  data,  V:  4 

Source  materials,  of  job  analysis 
data  and  job  information,  I: 
97-98 

of  training  courses  and  in- 
structional materials,  I: 
95-97 

Staff  training,  instructional, 

IV:  7-12 

Standards,  Job  Performaiice 
Measures,  I:  180-184 
task,  I:  28-33 
terminal  learning  objective 
test,  II;  47-50 
training,  Ii:  47-50 
State-of-the-art,  III:  120 
Steering  and  guiding;  learning 
sub-category  8,  III:  56-57, 

94-95 

Stimulus  criteria  (media).  III: 
107-108 

Storyboards,  III:  234,  239-242, 
245-246 

Student  managanent  plans,  see 
Management  plans,  student  and 
Instructional  management  plan 
Student  self-managed  instruction. 

Ill:  195,  215 
Students'  Guide,  III:  269 
Students'  manual,  IV:  6 
Subject  Matter  Experts  (SMEs), 

1:  79,  81-82,  133 


Suppletnent.'ry  instruction.  III:  2r/) 
250 

System  Master  Plan  (SMP),  III:  1C5, 
107,  138-142;  V:  10,  90-91 
System  revision,  V:  87-120 

based  on  changes  in  doctrine  or 
content  of  DOS,  V:  94 
based  on  efficiency  of  instruc- 
tion, V:  96-103 
based  on  External  Evaluation 
Report,  V:  95-96 
based  on  Internal  Evaluation 
Report,  V:  55 
description  of,  V:  87-89 
follow-up  activity,  V;  115 
for  improvement  of  i ns'' .’ucti  onal 
effectiveness,  V;  109-112 
for  operating  instruction,  V: 
105-106 

for  time  reduction,  V:  106-109 
guidelines  for  determining  revi- 
sion needs,  V:  103-112 
outputs,  V:  115-116 
preparation  of  revision  plan, 

V:  112-114 

priority  ratings  for,  V:  92 
procedures  for,  V:  93-115 
purposes  of,  V:  37-90,  93 
.sources  for,  V:  90-91 


Task(s),  I:  12 

checklist  task  inventory,  I: 
152-154 

clustering  of,  I:  244-245 
conditions  statement  for,  I;  24- 
26,  89-93 

delay  tolerance,  I:  123-125,  246 
guidelines  Tor  diagiarraning,  I: 

24 

inventory,  verifying  and  validat 
ing,  I:  66-68 

Job  Performance  Measures  for,  I: 
1 57-21 1 

multiple,  I:  154-165 
statements,  I:  13-17,  61 
unitary,  I:  ‘'64 

validated  task  list,  sample,  I: 
33,  86 

Tas';  selection,  I:  113-155 

criteria  for,  I:  118-133,  152- 
154 
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Task  selection  (continued) 
data  assessing  ancTcTecision- 
inakinq  for,  I:  '’39-148 
data  collection  for,  1:  134-136 
data  consolidation  ^"or,  I:  136- 
139 

management  constraints,  I;  143- 
144 

outputs,  I;  117,  148-149 
procedures,  I:  117-148 
rationale  for,  I;  HE -11? 
survey  sources,  I:  133-134 
Television  producer  (production 
personnel).  III:  229 
Television  program  productiori.  III: 
244 

Terminal  learning  objectives  (TLOs), 
II:  1-31 

action  statement  of,  II:  5-12 
conditions  stetement  of,  II:  5-6, 
9-10,  13-14 

guidelines  for  learner,  III:  4 
learning  analysis  for,  II:  18-30 
Harning  categories,  H:  ''6-17 
standards  statement  of,  II:  C-6, 
9-10,  14-i6 

Test  development,  II:  35-61 
attitudes  testing,  II;  45-46 
information  testing,  II:  42-44 
inputs,  II:  7 

ental  skills  testing,  II:  39-^2 
misclassifications,  il;  50-o2 
outputs , II:  37 , 60 
physical  "kills  testing,  II:  44-45 
procedures,  II:  38-60 
purposes,  II:  35 
rank-order  testing,  II:  52-55 
scoring,  II:  55-56 
standards,  II:  47-50 
types  of  tests,  II:  35,  38-39 
Testing  constraints  in  Job  Perform- 
ance Measures,  I:  166-177 
Trainer  Aopraisal  Kit  (TAK),  IV; 

7-12 

Trainer  Development  Program  (TRADER), 
IV:  9 

Training  Extension  Course  (TEC), 

Army,  III:  141-142 
Training  programs,  selection  of 
tasks  for,  I;  114-133,  139-148 
Training  task  categories,  see 
Learning  categories 


Trial  report  (small  group),  examp'ie, 
III:  340-3/’ 5 


Voice  cotunurii eating:  learning  sub- 
category 10,  HI:  63-65,  103- 
102 


Writer  {production  personnel),  III: 
229 


